[jira] [Issue Comment Deleted] (HDFS-15096) RBF : GetServerDefaults Should be Cached At Router
[
https://issues.apache.org/jira/browse/HDFS-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-15096:
Comment: was deleted
(was: | (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HDFS-15096 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-15096 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28614/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
)
> RBF : GetServerDefaults Should be Cached At Router
> --
>
> Key: HDFS-15096
> URL: https://issues.apache.org/jira/browse/HDFS-15096
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: GetServerDefault-Cached.PNG, HDFS-15096-01.patch,
> Server-Before.PNG
>
>
> Presently each getServerDefault calls are triggered to the Namespace, The
> DFSClient caches the getServerDefaults(), similarly the Router can also cache
> the getServerDefaults(), as router connects to multiple clients, this
> improves performance for subsequent calls.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-15094) RBF : Reuse ugi string in ConnectionPoolID
[
https://issues.apache.org/jira/browse/HDFS-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-15094:
Comment: was deleted
(was: | (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-15094 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-15094 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28615/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
)
> RBF : Reuse ugi string in ConnectionPoolID
> --
>
> Key: HDFS-15094
> URL: https://issues.apache.org/jira/browse/HDFS-15094
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15094-01.patch, HDFS-15094-02.patch,
> UGI_AFTER-01.PNG, UGI_BEFORE-01.PNG
>
>
> The connectionPoolID Hash Code and equals contains ugi.toString(), These
> methods are used as part of getConnection() in ConnectionManager as part of
> every call.
> The ugi.toString() eats up considerable amount of time, the Hash Calculation
> itself is ~10 percent of the total time of the call. And even more for the
> equals method.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15080) Fix the issue in reading persistent memory cache with an offset
[ https://issues.apache.org/jira/browse/HDFS-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Feilong He updated HDFS-15080: -- Description: Some applications can read a segment of pmem cache with an offset specified. The previous implementation for pmem cache read with DirectByteBuffer didn't cover this situation. Let me explain further. In our test, we used spark SQL to run some TPC-DS workload to read the cache data and hits read exception. This was due to the missed seek offset arg, which is used in spark SQL to read data packet by packet. was:Some applications can read a segment of pmem cache with an offset specified. The previous implementation for pmem cache read with DirectByteBuffer didn't cover this situation. > Fix the issue in reading persistent memory cache with an offset > --- > > Key: HDFS-15080 > URL: https://issues.apache.org/jira/browse/HDFS-15080 > Project: Hadoop HDFS > Issue Type: Bug > Components: caching, datanode >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: HDFS-15080-000.patch, HDFS-15080-branch-3.1-000.patch, > HDFS-15080-branch-3.2-000.patch > > > Some applications can read a segment of pmem cache with an offset specified. > The previous implementation for pmem cache read with DirectByteBuffer didn't > cover this situation. > Let me explain further. In our test, we used spark SQL to run some TPC-DS > workload to read the cache data and hits read exception. This was due to the > missed seek offset arg, which is used in spark SQL to read data packet by > packet. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15094) RBF : Reuse ugi string in ConnectionPoolID
[
https://issues.apache.org/jira/browse/HDFS-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010388#comment-17010388
]
Hadoop QA commented on HDFS-15094:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-15094 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-15094 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28615/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF : Reuse ugi string in ConnectionPoolID
> --
>
> Key: HDFS-15094
> URL: https://issues.apache.org/jira/browse/HDFS-15094
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15094-01.patch, HDFS-15094-02.patch,
> UGI_AFTER-01.PNG, UGI_BEFORE-01.PNG
>
>
> The connectionPoolID Hash Code and equals contains ugi.toString(), These
> methods are used as part of getConnection() in ConnectionManager as part of
> every call.
> The ugi.toString() eats up considerable amount of time, the Hash Calculation
> itself is ~10 percent of the total time of the call. And even more for the
> equals method.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15094) RBF : Reuse ugi string in ConnectionPoolID
[ https://issues.apache.org/jira/browse/HDFS-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15094: Attachment: UGI_AFTER-01.PNG > RBF : Reuse ugi string in ConnectionPoolID > -- > > Key: HDFS-15094 > URL: https://issues.apache.org/jira/browse/HDFS-15094 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15094-01.patch, HDFS-15094-02.patch, > UGI_AFTER-01.PNG, UGI_BEFORE-01.PNG > > > The connectionPoolID Hash Code and equals contains ugi.toString(), These > methods are used as part of getConnection() in ConnectionManager as part of > every call. > The ugi.toString() eats up considerable amount of time, the Hash Calculation > itself is ~10 percent of the total time of the call. And even more for the > equals method. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15094) RBF : Reuse ugi string in ConnectionPoolID
[ https://issues.apache.org/jira/browse/HDFS-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010386#comment-17010386 ] Ayush Saxena commented on HDFS-15094: - Sharing Profiler Outputs : Before : !UGI_BEFORE-01.PNG! After : !UGI_AFTER-01.PNG! > RBF : Reuse ugi string in ConnectionPoolID > -- > > Key: HDFS-15094 > URL: https://issues.apache.org/jira/browse/HDFS-15094 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15094-01.patch, HDFS-15094-02.patch, > UGI_AFTER-01.PNG, UGI_BEFORE-01.PNG > > > The connectionPoolID Hash Code and equals contains ugi.toString(), These > methods are used as part of getConnection() in ConnectionManager as part of > every call. > The ugi.toString() eats up considerable amount of time, the Hash Calculation > itself is ~10 percent of the total time of the call. And even more for the > equals method. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15094) RBF : Reuse ugi string in ConnectionPoolID
[ https://issues.apache.org/jira/browse/HDFS-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15094: Attachment: UGI_BEFORE-01.PNG > RBF : Reuse ugi string in ConnectionPoolID > -- > > Key: HDFS-15094 > URL: https://issues.apache.org/jira/browse/HDFS-15094 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15094-01.patch, HDFS-15094-02.patch, > UGI_AFTER-01.PNG, UGI_BEFORE-01.PNG > > > The connectionPoolID Hash Code and equals contains ugi.toString(), These > methods are used as part of getConnection() in ConnectionManager as part of > every call. > The ugi.toString() eats up considerable amount of time, the Hash Calculation > itself is ~10 percent of the total time of the call. And even more for the > equals method. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15096) RBF : GetServerDefaults Should be Cached At Router
[
https://issues.apache.org/jira/browse/HDFS-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010380#comment-17010380
]
Hadoop QA commented on HDFS-15096:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HDFS-15096 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-15096 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28614/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF : GetServerDefaults Should be Cached At Router
> --
>
> Key: HDFS-15096
> URL: https://issues.apache.org/jira/browse/HDFS-15096
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: GetServerDefault-Cached.PNG, HDFS-15096-01.patch,
> Server-Before.PNG
>
>
> Presently each getServerDefault calls are triggered to the Namespace, The
> DFSClient caches the getServerDefaults(), similarly the Router can also cache
> the getServerDefaults(), as router connects to multiple clients, this
> improves performance for subsequent calls.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15096) RBF : GetServerDefaults Should be Cached At Router
[ https://issues.apache.org/jira/browse/HDFS-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010376#comment-17010376 ] Ayush Saxena commented on HDFS-15096: - Tested locally, Sharing the profiler times, Well that logically it would be less, but still since I had, Sharing it : Before : !Server-Before.PNG! After : !GetServerDefault-Cached.PNG! Will add comments as said latter today and raise a JIRA for using timeperiod instead millis > RBF : GetServerDefaults Should be Cached At Router > -- > > Key: HDFS-15096 > URL: https://issues.apache.org/jira/browse/HDFS-15096 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: GetServerDefault-Cached.PNG, HDFS-15096-01.patch, > Server-Before.PNG > > > Presently each getServerDefault calls are triggered to the Namespace, The > DFSClient caches the getServerDefaults(), similarly the Router can also cache > the getServerDefaults(), as router connects to multiple clients, this > improves performance for subsequent calls. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15096) RBF : GetServerDefaults Should be Cached At Router
[ https://issues.apache.org/jira/browse/HDFS-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15096: Attachment: (was: GetServerDefault-Cached.PNG) > RBF : GetServerDefaults Should be Cached At Router > -- > > Key: HDFS-15096 > URL: https://issues.apache.org/jira/browse/HDFS-15096 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: GetServerDefault-Cached.PNG, HDFS-15096-01.patch, > Server-Before.PNG > > > Presently each getServerDefault calls are triggered to the Namespace, The > DFSClient caches the getServerDefaults(), similarly the Router can also cache > the getServerDefaults(), as router connects to multiple clients, this > improves performance for subsequent calls. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15096) RBF : GetServerDefaults Should be Cached At Router
[ https://issues.apache.org/jira/browse/HDFS-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15096: Attachment: GetServerDefault-Cached.PNG > RBF : GetServerDefaults Should be Cached At Router > -- > > Key: HDFS-15096 > URL: https://issues.apache.org/jira/browse/HDFS-15096 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: GetServerDefault-Cached.PNG, > GetServerDefault-Cached.PNG, HDFS-15096-01.patch, Server-Before.PNG > > > Presently each getServerDefault calls are triggered to the Namespace, The > DFSClient caches the getServerDefaults(), similarly the Router can also cache > the getServerDefaults(), as router connects to multiple clients, this > improves performance for subsequent calls. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15096) RBF : GetServerDefaults Should be Cached At Router
[ https://issues.apache.org/jira/browse/HDFS-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15096: Attachment: GetServerDefault-Cached.PNG > RBF : GetServerDefaults Should be Cached At Router > -- > > Key: HDFS-15096 > URL: https://issues.apache.org/jira/browse/HDFS-15096 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: GetServerDefault-Cached.PNG, > GetServerDefault-Cached.PNG, HDFS-15096-01.patch, Server-Before.PNG > > > Presently each getServerDefault calls are triggered to the Namespace, The > DFSClient caches the getServerDefaults(), similarly the Router can also cache > the getServerDefaults(), as router connects to multiple clients, this > improves performance for subsequent calls. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15096) RBF : GetServerDefaults Should be Cached At Router
[ https://issues.apache.org/jira/browse/HDFS-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15096: Attachment: Server-Before.PNG > RBF : GetServerDefaults Should be Cached At Router > -- > > Key: HDFS-15096 > URL: https://issues.apache.org/jira/browse/HDFS-15096 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15096-01.patch, Server-Before.PNG > > > Presently each getServerDefault calls are triggered to the Namespace, The > DFSClient caches the getServerDefaults(), similarly the Router can also cache > the getServerDefaults(), as router connects to multiple clients, this > improves performance for subsequent calls. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15098) Add SM4 encryption method for HDFS
[
https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010357#comment-17010357
]
Hadoop QA commented on HDFS-15098:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
5s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
22m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 20m 24s{color} |
{color:red} root generated 9 new + 21 unchanged - 5 fixed = 30 total (was 26)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 35s{color} | {color:orange} root: The patch generated 19 new + 179 unchanged
- 3 fixed = 198 total (was 182) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 29s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
1s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestFixKerberosTicketOrder |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HDFS-15098 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12990156/HDFS-15098.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle cc |
| uname | Linux 255b13d12151 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
[jira] [Updated] (HDFS-15072) HDFS MiniCluster fails to start when run in directory path with a %
[ https://issues.apache.org/jira/browse/HDFS-15072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-15072: - Tags: (was: beginner) > HDFS MiniCluster fails to start when run in directory path with a % > --- > > Key: HDFS-15072 > URL: https://issues.apache.org/jira/browse/HDFS-15072 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.5, 3.3.0 > Environment: I encountered this on a Mac while running an HBase > minicluster that was using Hadoop 2.7.5. However, the code looks the same in > trunk so it likely affects most or all current versions. >Reporter: Geoffrey Jacoby >Assignee: Masatake Iwasaki >Priority: Minor > Fix For: 3.3.0, 3.1.4, 3.2.2 > > > FSVolumeImpl.initializeCacheExecutor calls Guava's ThreadPoolExecutorBuilder. > setNameFormat, passing in the String representation of the parent File. Guava > will take the String whole and pass it to String.format, which uses % as a > special character. That means that if parent.toString() contains a percentage > sign, followed by a character that's illegal to use as a formatter in > String.format(), you'll get an exception that stops the MiniCluster from > starting up. > I did not check to see if this would also happen on a normal DataNode daemon. > initializeCacheExecutor should escape the parent file name before passing it > in. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15072) HDFS MiniCluster fails to start when run in directory path with a %
[ https://issues.apache.org/jira/browse/HDFS-15072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-15072: - Committed to trunk, branch-3.2, and branch-3.1. Thanks [~iwasakims]. > HDFS MiniCluster fails to start when run in directory path with a % > --- > > Key: HDFS-15072 > URL: https://issues.apache.org/jira/browse/HDFS-15072 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.5, 3.3.0 > Environment: I encountered this on a Mac while running an HBase > minicluster that was using Hadoop 2.7.5. However, the code looks the same in > trunk so it likely affects most or all current versions. >Reporter: Geoffrey Jacoby >Assignee: Masatake Iwasaki >Priority: Minor > Fix For: 3.3.0, 3.1.4, 3.2.2 > > > FSVolumeImpl.initializeCacheExecutor calls Guava's ThreadPoolExecutorBuilder. > setNameFormat, passing in the String representation of the parent File. Guava > will take the String whole and pass it to String.format, which uses % as a > special character. That means that if parent.toString() contains a percentage > sign, followed by a character that's illegal to use as a formatter in > String.format(), you'll get an exception that stops the MiniCluster from > starting up. > I did not check to see if this would also happen on a normal DataNode daemon. > initializeCacheExecutor should escape the parent file name before passing it > in. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15072) HDFS MiniCluster fails to start when run in directory path with a %
[ https://issues.apache.org/jira/browse/HDFS-15072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-15072: - Fix Version/s: 3.2.2 3.1.4 > HDFS MiniCluster fails to start when run in directory path with a % > --- > > Key: HDFS-15072 > URL: https://issues.apache.org/jira/browse/HDFS-15072 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.5, 3.3.0 > Environment: I encountered this on a Mac while running an HBase > minicluster that was using Hadoop 2.7.5. However, the code looks the same in > trunk so it likely affects most or all current versions. >Reporter: Geoffrey Jacoby >Assignee: Masatake Iwasaki >Priority: Minor > Fix For: 3.3.0, 3.1.4, 3.2.2 > > > FSVolumeImpl.initializeCacheExecutor calls Guava's ThreadPoolExecutorBuilder. > setNameFormat, passing in the String representation of the parent File. Guava > will take the String whole and pass it to String.format, which uses % as a > special character. That means that if parent.toString() contains a percentage > sign, followed by a character that's illegal to use as a formatter in > String.format(), you'll get an exception that stops the MiniCluster from > starting up. > I did not check to see if this would also happen on a normal DataNode daemon. > initializeCacheExecutor should escape the parent file name before passing it > in. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15072) HDFS MiniCluster fails to start when run in directory path with a %
[ https://issues.apache.org/jira/browse/HDFS-15072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-15072: Fix Version/s: 3.3.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for reviewing this, [~aajisaka]. > HDFS MiniCluster fails to start when run in directory path with a % > --- > > Key: HDFS-15072 > URL: https://issues.apache.org/jira/browse/HDFS-15072 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.5, 3.3.0 > Environment: I encountered this on a Mac while running an HBase > minicluster that was using Hadoop 2.7.5. However, the code looks the same in > trunk so it likely affects most or all current versions. >Reporter: Geoffrey Jacoby >Assignee: Masatake Iwasaki >Priority: Minor > Fix For: 3.3.0 > > > FSVolumeImpl.initializeCacheExecutor calls Guava's ThreadPoolExecutorBuilder. > setNameFormat, passing in the String representation of the parent File. Guava > will take the String whole and pass it to String.format, which uses % as a > special character. That means that if parent.toString() contains a percentage > sign, followed by a character that's illegal to use as a formatter in > String.format(), you'll get an exception that stops the MiniCluster from > starting up. > I did not check to see if this would also happen on a normal DataNode daemon. > initializeCacheExecutor should escape the parent file name before passing it > in. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15087) RBF: Balance/Rename across federation namespaces
[ https://issues.apache.org/jira/browse/HDFS-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009739#comment-17009739 ] Jinglun edited comment on HDFS-15087 at 1/8/20 2:48 AM: Hi [~linyiqun], thanks your suggestion ! I prepared a initial patch based on xiaomi branch. There are few annotation in the patch. So I write some annotations below for the important classes. * FRHelper, FederationRenameV2, FederationRenameV1(Deprecated)：Serialization & Deserialization. * FSFedRenameV2Op: It handles saveSubTree and graftSubTree rpcs. Like FSDirRenameOp. * BasicInfo: It contains the basic information of an HFR job. * BlockLinker: It does all the hard links. Including collecting and hardlink. * BlocksToDup: Blocks in batch. * FsDatasetImpl: method addBlocksToNewPool is added to hard link replicas. * IDConsumer: It's used by NameNode. After the Ids are pre-allocated, we use IDConsumer to consume all the allocated Ids. * LockHelper: It helps release and restore all the write locks. * ParallelConsumer: The super class of ChecksumDirectoriesChecker. * ChecksumDirectoriesChecker: After the HardLink phase done, It verifies the source path and destination path. * JobScheduler, Job, JobContext: The Scheduler model in the design doc. * FederationRenameProcedure: It's the HFR. * DistCpProcedure: Another version of HFR. Use distcp instead of saveSubTree + graftSubTree + HardLink. was (Author: lijinglun): Hi [~linyiqun], thanks your suggestion ! I prepared a initial patch based on xiaomi branch. There are few annotation in the patch. So I write some annotations below for the important classes. * FRHelper, FederationRenameV2, FederationRenameV1(Deprecated)：Serialization & Deserialization. * FSFedRenameV2Op: It handles saveSubTree and graftSubTree rpcs. Like FSDirRenameOp. * BasicInfo: It contains the basic information of an HFR job. * BlockLinker: It does all the hard links. Including collecting and hardlink. * BlocksToDup: Blocks in batch. * FsDatasetImpl: method addBlocksToNewPool is added to hard link replicas. * IDConsumer: It's used by NameNode. After the Ids are pre-allocated, we use IDConsumer to consume all the allocated Ids. * LockHelper: It helps release and restore all the write locks. * ParallelConsumer: The super class of ChecksumDirectoriesChecker. * ChecksumDirectoriesChecker: After the HardLink phase done, It verifies the source path and destination path. * JobScheduler, Job, JobContext: The Scheduler model in the design doc. * FederationRenameProcedure: It's the HFR. * DistCpProcedure: Use distcp instead of saveSubTree + graftSubTree + HardLink. > RBF: Balance/Rename across federation namespaces > > > Key: HDFS-15087 > URL: https://issues.apache.org/jira/browse/HDFS-15087 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Priority: Major > Attachments: HDFS-15087.initial.patch, HFR_Rename Across Federation > Namespaces.pdf > > > The Xiaomi storage team has developed a new feature called HFR(HDFS > Federation Rename) that enables us to do balance/rename across federation > namespaces. The idea is to first move the meta to the dst NameNode and then > link all the replicas. It has been working in our largest production cluster > for 2 months. We use it to balance the namespaces. It turns out HFR is fast > and flexible. The detail could be found in the design doc. > Looking forward to a lively discussion. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15072) HDFS MiniCluster fails to start when run in directory path with a %
[ https://issues.apache.org/jira/browse/HDFS-15072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010290#comment-17010290 ] Hudson commented on HDFS-15072: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17825 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17825/]) HDFS-15072. HDFS MiniCluster fails to start when run in directory path (aajisaka: rev a43c177f1d4c2b6149a2680dd23d91103eca3be0) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java > HDFS MiniCluster fails to start when run in directory path with a % > --- > > Key: HDFS-15072 > URL: https://issues.apache.org/jira/browse/HDFS-15072 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.5, 3.3.0 > Environment: I encountered this on a Mac while running an HBase > minicluster that was using Hadoop 2.7.5. However, the code looks the same in > trunk so it likely affects most or all current versions. >Reporter: Geoffrey Jacoby >Assignee: Masatake Iwasaki >Priority: Minor > > FSVolumeImpl.initializeCacheExecutor calls Guava's ThreadPoolExecutorBuilder. > setNameFormat, passing in the String representation of the parent File. Guava > will take the String whole and pass it to String.format, which uses % as a > special character. That means that if parent.toString() contains a percentage > sign, followed by a character that's illegal to use as a formatter in > String.format(), you'll get an exception that stops the MiniCluster from > starting up. > I did not check to see if this would also happen on a normal DataNode daemon. > initializeCacheExecutor should escape the parent file name before passing it > in. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15099) [SBN Read] getBlockLocations() should throw ObserverRetryOnActiveException on an attempt to change aTime on ObserverNode
[
https://issues.apache.org/jira/browse/HDFS-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010284#comment-17010284
]
Konstantin Shvachko edited comment on HDFS-15099 at 1/8/20 2:41 AM:
Currently {{getBlockLocations()}} trying to update aTime on Observer only logs
a warning {{StandbyException - "Operation category WRITE is not supported in
state observer"}}, but doesn't redirect to Active.
We see when the read load is high this correlates with increase in RPC queue
time on Observer, which slows down the cluster in extreme cases.
was (Author: shv):
Currently {{getBlockLocations()}} on Observer only logs a warning
{{StandbyException - "Operation category WRITE is not supported in state
observer"}}, but doesn't redirect to Active.
We see when the read load is high this correlates with increase in RPC queue
time on Observer, which slows down the cluster in extreme cases.
> [SBN Read] getBlockLocations() should throw ObserverRetryOnActiveException on
> an attempt to change aTime on ObserverNode
>
>
> Key: HDFS-15099
> URL: https://issues.apache.org/jira/browse/HDFS-15099
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.10.0
>Reporter: Konstantin Shvachko
>Priority: Major
>
> The precision of updating an INode's aTime while executing
> {{getBlockLocations()}} is 1 hour by default. Updates cannot be handled by
> ObserverNode, so the call should be redirected to Active NameNode. In order
> to redirect to active the ObserverNode should through
> {{ObserverRetryOnActiveException}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15099) [SBN Read] getBlockLocations() should throw ObserverRetryOnActiveException on an attempt to change aTime on ObserverNode
[
https://issues.apache.org/jira/browse/HDFS-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010284#comment-17010284
]
Konstantin Shvachko commented on HDFS-15099:
Currently {{getBlockLocations()}} on Observer only logs a warning
{{StandbyException - "Operation category WRITE is not supported in state
observer"}}, but doesn't redirect to Active.
We see when the read load is high this correlates with increase in RPC queue
time on Observer, which slows down the cluster in extreme cases.
> [SBN Read] getBlockLocations() should throw ObserverRetryOnActiveException on
> an attempt to change aTime on ObserverNode
>
>
> Key: HDFS-15099
> URL: https://issues.apache.org/jira/browse/HDFS-15099
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.10.0
>Reporter: Konstantin Shvachko
>Priority: Major
>
> The precision of updating an INode's aTime while executing
> {{getBlockLocations()}} is 1 hour by default. Updates cannot be handled by
> ObserverNode, so the call should be redirected to Active NameNode. In order
> to redirect to active the ObserverNode should through
> {{ObserverRetryOnActiveException}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15099) [SBN Read] getBlockLocations() should throw ObserverRetryOnActiveException on an attempt to change aTime on ObserverNode
Konstantin Shvachko created HDFS-15099:
--
Summary: [SBN Read] getBlockLocations() should throw
ObserverRetryOnActiveException on an attempt to change aTime on ObserverNode
Key: HDFS-15099
URL: https://issues.apache.org/jira/browse/HDFS-15099
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.10.0
Reporter: Konstantin Shvachko
The precision of updating an INode's aTime while executing
{{getBlockLocations()}} is 1 hour by default. Updates cannot be handled by
ObserverNode, so the call should be redirected to Active NameNode. In order to
redirect to active the ObserverNode should through
{{ObserverRetryOnActiveException}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liusheng updated HDFS-15098: Attachment: HDFS-15098.001.patch Status: Patch Available (was: Open) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: liusheng >Priority: Major > Attachments: HDFS-15098.001.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15098) Add SM4 encryption method for HDFS
liusheng created HDFS-15098: --- Summary: Add SM4 encryption method for HDFS Key: HDFS-15098 URL: https://issues.apache.org/jira/browse/HDFS-15098 Project: Hadoop HDFS Issue Type: Bug Reporter: liusheng SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far been rejected by ISO. One of the reasons for the rejection has been opposition to the WAPI fast-track proposal by the IEEE. please see: [https://en.wikipedia.org/wiki/SM4_(cipher)] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15096) RBF : GetServerDefaults Should be Cached At Router
[ https://issues.apache.org/jira/browse/HDFS-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010138#comment-17010138 ] Íñigo Goiri commented on HDFS-15096: I was thinking on veryfying that the NameNode method was only called once within the caching period. It might be a little overkill but I leave it up to you. > RBF : GetServerDefaults Should be Cached At Router > -- > > Key: HDFS-15096 > URL: https://issues.apache.org/jira/browse/HDFS-15096 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15096-01.patch > > > Presently each getServerDefault calls are triggered to the Namespace, The > DFSClient caches the getServerDefaults(), similarly the Router can also cache > the getServerDefaults(), as router connects to multiple clients, this > improves performance for subsequent calls. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15095) Fix accidental comment in flaky test TestDecommissioningStatus
[
https://issues.apache.org/jira/browse/HDFS-15095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010110#comment-17010110
]
Jim Brennan edited comment on HDFS-15095 at 1/7/20 9:41 PM:
Thanks for the patch [~ahussein]!
I am +1 (non-binding) on patch 002. Looks good to me!
was (Author: jim_brennan):
Thanks for the patch [~ahussein]! I am +1 (non-binding) on patch 002. Looks
good to me!
> Fix accidental comment in flaky test TestDecommissioningStatus
> --
>
> Key: HDFS-15095
> URL: https://issues.apache.org/jira/browse/HDFS-15095
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: HDFS-15095.001.patch, HDFS-15095.002.patch
>
>
> There are some old Jiras suggesting that "{{testDecommissionStatus"}} is
> flaky.
> * HDFS-12188
> * HDFS-9599
> * HDFS-9950
> * HDFS-10755
> However, HDFS-14854 fix accidentally commented out one of the checks in
> {{TestDecommissioningStatus.testDecommissionStatus()"}}. This Jira will
> restore the commented out code and adds a blocking queue to make the test
> case deterministic.
> My intuition is that monitor task launched by AdminManager may not have
> enough time to act before we start verifying the status. I suggest the force
> the main thread to block until the node is added to the blocked node.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15095) Fix accidental comment in flaky test TestDecommissioningStatus
[
https://issues.apache.org/jira/browse/HDFS-15095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010110#comment-17010110
]
Jim Brennan commented on HDFS-15095:
Thanks for the patch [~ahussein]! I am +1 (non-binding) on patch 002. Looks
good to me!
> Fix accidental comment in flaky test TestDecommissioningStatus
> --
>
> Key: HDFS-15095
> URL: https://issues.apache.org/jira/browse/HDFS-15095
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: HDFS-15095.001.patch, HDFS-15095.002.patch
>
>
> There are some old Jiras suggesting that "{{testDecommissionStatus"}} is
> flaky.
> * HDFS-12188
> * HDFS-9599
> * HDFS-9950
> * HDFS-10755
> However, HDFS-14854 fix accidentally commented out one of the checks in
> {{TestDecommissioningStatus.testDecommissionStatus()"}}. This Jira will
> restore the commented out code and adds a blocking queue to make the test
> case deterministic.
> My intuition is that monitor task launched by AdminManager may not have
> enough time to act before we start verifying the status. I suggest the force
> the main thread to block until the node is added to the blocked node.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15094) RBF : Reuse ugi string in ConnectionPoolID
[
https://issues.apache.org/jira/browse/HDFS-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010099#comment-17010099
]
Hadoop QA commented on HDFS-15094:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
4s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 30s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRouterClientRejectOverload |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HDFS-15094 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12990128/HDFS-15094-02.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f411d561800e 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d1f5976 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_232 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/28612/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28612/testReport/ |
| Max. process+thread count | 2571 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
[jira] [Commented] (HDFS-15094) RBF : Reuse ugi string in ConnectionPoolID
[ https://issues.apache.org/jira/browse/HDFS-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010059#comment-17010059 ] Ayush Saxena commented on HDFS-15094: - if in that ugi.equals method instead of doing == for subject, if we call equals for the subject, then we could have used it but doing that seems not very safe, Discussed internally regarding tweaking the ugi class this way, but... > RBF : Reuse ugi string in ConnectionPoolID > -- > > Key: HDFS-15094 > URL: https://issues.apache.org/jira/browse/HDFS-15094 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15094-01.patch, HDFS-15094-02.patch > > > The connectionPoolID Hash Code and equals contains ugi.toString(), These > methods are used as part of getConnection() in ConnectionManager as part of > every call. > The ugi.toString() eats up considerable amount of time, the Hash Calculation > itself is ~10 percent of the total time of the call. And even more for the > equals method. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15094) RBF : Reuse ugi string in ConnectionPoolID
[ https://issues.apache.org/jira/browse/HDFS-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15094: Attachment: HDFS-15094-02.patch > RBF : Reuse ugi string in ConnectionPoolID > -- > > Key: HDFS-15094 > URL: https://issues.apache.org/jira/browse/HDFS-15094 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15094-01.patch, HDFS-15094-02.patch > > > The connectionPoolID Hash Code and equals contains ugi.toString(), These > methods are used as part of getConnection() in ConnectionManager as part of > every call. > The ugi.toString() eats up considerable amount of time, the Hash Calculation > itself is ~10 percent of the total time of the call. And even more for the > equals method. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15096) RBF : GetServerDefaults Should be Cached At Router
[ https://issues.apache.org/jira/browse/HDFS-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010034#comment-17010034 ] Ayush Saxena commented on HDFS-15096: - Thanx [~elgoiri] for the review. What should I verify in the test? Should I call getServerDefaults() multiple times? Since it is just caching, so output would be same only. > RBF : GetServerDefaults Should be Cached At Router > -- > > Key: HDFS-15096 > URL: https://issues.apache.org/jira/browse/HDFS-15096 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15096-01.patch > > > Presently each getServerDefault calls are triggered to the Namespace, The > DFSClient caches the getServerDefaults(), similarly the Router can also cache > the getServerDefaults(), as router connects to multiple clients, this > improves performance for subsequent calls. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14578) AvailableSpaceBlockPlacementPolicy always prefers local node
[
https://issues.apache.org/jira/browse/HDFS-14578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009953#comment-17009953
]
Ayush Saxena commented on HDFS-14578:
-
TestHdfsConfigFields is related, Findbugs has concerns but I don't think so a
valid one, would add it to exclude this. Storage types is being used by calling
method
> AvailableSpaceBlockPlacementPolicy always prefers local node
>
>
> Key: HDFS-14578
> URL: https://issues.apache.org/jira/browse/HDFS-14578
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
>Affects Versions: 2.8.0, 2.7.4, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14578-WIP-01.patch, HDFS-14758-01.patch
>
>
> It looks like AvailableSpaceBlockPlacementPolicy prefers local disk just like
> in the BlockPlacementPolicyDefault
>
> As Yongjun mentioned in
> [HDFS-8131|https://issues.apache.org/jira/browse/HDFS-8131?focusedCommentId=16558739=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16558739],
>
> {quote}Class AvailableSpaceBlockPlacementPolicy extends
> BlockPlacementPolicyDefault. But it doesn't change the behavior of choosing
> the first node in BlockPlacementPolicyDefault, so even with this new feature,
> the local DN is always chosen as the first DN (of course when it is not
> excluded), and the new feature only changes the selection of the rest of the
> two DNs.
> {quote}
> I'm file this Jira as I groom Cloudera's internal Jira and found this
> unreported issue. We do have a customer hitting this problem. I don't have a
> fix, but thought it would be beneficial to report it to Apache Jira.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14578) AvailableSpaceBlockPlacementPolicy always prefers local node
[
https://issues.apache.org/jira/browse/HDFS-14578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009945#comment-17009945
]
Hadoop QA commented on HDFS-14578:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 48s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 14 new + 459 unchanged - 0 fixed = 473 total (was 459) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 38s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
30s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}115m 14s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Dead store to storageTypes in
org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.chooseLocalStorage(Node,
Set, long, int, List, boolean, EnumMap, boolean) At
AvailableSpaceBlockPlacementPolicy.java:org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.chooseLocalStorage(Node,
Set, long, int, List, boolean, EnumMap, boolean) At
AvailableSpaceBlockPlacementPolicy.java:[line 136] |
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.server.namenode.TestRedudantBlocks |
| | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HDFS-14578 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12990107/HDFS-14758-01.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5262f5ef12cf 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Commented] (HDFS-15095) Fix accidental comment in flaky test TestDecommissioningStatus
[
https://issues.apache.org/jira/browse/HDFS-15095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009940#comment-17009940
]
Ahmed Hussein commented on HDFS-15095:
--
[~Jim_Brennan] and [~kihwal] Can you please take a look that patch? the
failures of the Junit tests seem to come from flaky tests unrelated to the
change. There are some couple of Jiras related to them.
* HDFS-15092
* HDFS-13040
* HDFS-10644
> Fix accidental comment in flaky test TestDecommissioningStatus
> --
>
> Key: HDFS-15095
> URL: https://issues.apache.org/jira/browse/HDFS-15095
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: HDFS-15095.001.patch, HDFS-15095.002.patch
>
>
> There are some old Jiras suggesting that "{{testDecommissionStatus"}} is
> flaky.
> * HDFS-12188
> * HDFS-9599
> * HDFS-9950
> * HDFS-10755
> However, HDFS-14854 fix accidentally commented out one of the checks in
> {{TestDecommissioningStatus.testDecommissionStatus()"}}. This Jira will
> restore the commented out code and adds a blocking queue to make the test
> case deterministic.
> My intuition is that monitor task launched by AdminManager may not have
> enough time to act before we start verifying the status. I suggest the force
> the main thread to block until the node is added to the blocked node.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15095) Fix accidental comment in flaky test TestDecommissioningStatus
[
https://issues.apache.org/jira/browse/HDFS-15095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009935#comment-17009935
]
Ahmed Hussein commented on HDFS-15095:
--
Stack trace of failed tests from second submission:
*1. TestRedudantBlocks.testProcessOverReplicatedAndRedudantBlock*
{code:bash}
[INFO] Running org.apache.hadoop.hdfs.server.namenode.TestRedudantBlocks
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.046
s <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.TestRedudantBlocks
[ERROR]
testProcessOverReplicatedAndRedudantBlock(org.apache.hadoop.hdfs.server.namenode.TestRedudantBlocks)
Time elapsed: 9.966 s <<< FAILURE!
java.lang.AssertionError: expected:<5> but was:<4>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at
org.apache.hadoop.hdfs.server.namenode.TestRedudantBlocks.testProcessOverReplicatedAndRedudantBlock(TestRedudantBlocks.java:138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{code}
*2. TestBlockStatsMXBean.testStorageTypeStatsWhenStorageFailed*
{code:bash}
[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 25.87 s
<<< FAILURE! - in
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
[ERROR]
testStorageTypeStatsWhenStorageFailed(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean)
Time elapsed: 16.192 s <<< FAILURE!
java.lang.AssertionError: expected:<6> but was:<3>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean.testStorageTypeStatsWhenStorageFailed(TestBlockStatsMXBean.java:213)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
[jira] [Commented] (HDFS-15094) RBF : Reuse ugi string in ConnectionPoolID
[ https://issues.apache.org/jira/browse/HDFS-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009919#comment-17009919 ] Íñigo Goiri commented on HDFS-15094: Let's add a comment explaining this operation is expensive and that we need to cache it. Could you share a screenshot with the cost difference? BTW, I'm surprise that ugi doesn't have an efficient hash and equals method. > RBF : Reuse ugi string in ConnectionPoolID > -- > > Key: HDFS-15094 > URL: https://issues.apache.org/jira/browse/HDFS-15094 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15094-01.patch > > > The connectionPoolID Hash Code and equals contains ugi.toString(), These > methods are used as part of getConnection() in ConnectionManager as part of > every call. > The ugi.toString() eats up considerable amount of time, the Hash Calculation > itself is ~10 percent of the total time of the call. And even more for the > equals method. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15096) RBF : GetServerDefaults Should be Cached At Router
[ https://issues.apache.org/jira/browse/HDFS-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009916#comment-17009916 ] Íñigo Goiri commented on HDFS-15096: * can we make the config use time period instead of Millis? Probably we should do that in a separate JIRA. * I think we could have a unit test. * can we add a comment where we do the caching to clarify why? > RBF : GetServerDefaults Should be Cached At Router > -- > > Key: HDFS-15096 > URL: https://issues.apache.org/jira/browse/HDFS-15096 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15096-01.patch > > > Presently each getServerDefault calls are triggered to the Namespace, The > DFSClient caches the getServerDefaults(), similarly the Router can also cache > the getServerDefaults(), as router connects to multiple clients, this > improves performance for subsequent calls. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15096) RBF : GetServerDefaults Should be Cached At Router
[
https://issues.apache.org/jira/browse/HDFS-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009832#comment-17009832
]
Hadoop QA commented on HDFS-15096:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 45s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
29s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HDFS-15096 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12990098/HDFS-15096-01.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux dddee707a294 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bc366d4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_232 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28610/testReport/ |
| Max. process+thread count | 3021 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28610/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF : GetServerDefaults Should be Cached At
[jira] [Commented] (HDFS-15090) RBF: MountPoint Listing Should Return Flag Values Of Destination
[ https://issues.apache.org/jira/browse/HDFS-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009798#comment-17009798 ] Ayush Saxena commented on HDFS-15090: - Thanx [~tasanuma] for the review and commit. :) > RBF: MountPoint Listing Should Return Flag Values Of Destination > > > Key: HDFS-15090 > URL: https://issues.apache.org/jira/browse/HDFS-15090 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-15090-01.patch > > > While doing listing, if a mount point is there and if the actual destination > is there, the owner and group details are taken from destination, similarly > the flags value can also be returned from destination. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14578) AvailableSpaceBlockPlacementPolicy always prefers local node
[
https://issues.apache.org/jira/browse/HDFS-14578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009794#comment-17009794
]
Ayush Saxena commented on HDFS-14578:
-
Thanx [~vinayakumarb] for the review, Have updated the patch, keeping the
default behavior as false.
Pls Review!!!
> AvailableSpaceBlockPlacementPolicy always prefers local node
>
>
> Key: HDFS-14578
> URL: https://issues.apache.org/jira/browse/HDFS-14578
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
>Affects Versions: 2.8.0, 2.7.4, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14578-WIP-01.patch, HDFS-14758-01.patch
>
>
> It looks like AvailableSpaceBlockPlacementPolicy prefers local disk just like
> in the BlockPlacementPolicyDefault
>
> As Yongjun mentioned in
> [HDFS-8131|https://issues.apache.org/jira/browse/HDFS-8131?focusedCommentId=16558739=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16558739],
>
> {quote}Class AvailableSpaceBlockPlacementPolicy extends
> BlockPlacementPolicyDefault. But it doesn't change the behavior of choosing
> the first node in BlockPlacementPolicyDefault, so even with this new feature,
> the local DN is always chosen as the first DN (of course when it is not
> excluded), and the new feature only changes the selection of the rest of the
> two DNs.
> {quote}
> I'm file this Jira as I groom Cloudera's internal Jira and found this
> unreported issue. We do have a customer hitting this problem. I don't have a
> fix, but thought it would be beneficial to report it to Apache Jira.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15094) RBF : Reuse ugi string in ConnectionPoolID
[
https://issues.apache.org/jira/browse/HDFS-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009795#comment-17009795
]
Hadoop QA commented on HDFS-15094:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 40s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 24s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRouterFaultTolerant |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HDFS-15094 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12990096/HDFS-15094-01.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5b5a2a48b007 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bc366d4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_232 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/28609/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28609/testReport/ |
| Max. process+thread count | 2780 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
|
[jira] [Updated] (HDFS-14578) AvailableSpaceBlockPlacementPolicy always prefers local node
[
https://issues.apache.org/jira/browse/HDFS-14578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-14578:
Attachment: HDFS-14758-01.patch
> AvailableSpaceBlockPlacementPolicy always prefers local node
>
>
> Key: HDFS-14578
> URL: https://issues.apache.org/jira/browse/HDFS-14578
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
>Affects Versions: 2.8.0, 2.7.4, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14578-WIP-01.patch, HDFS-14758-01.patch
>
>
> It looks like AvailableSpaceBlockPlacementPolicy prefers local disk just like
> in the BlockPlacementPolicyDefault
>
> As Yongjun mentioned in
> [HDFS-8131|https://issues.apache.org/jira/browse/HDFS-8131?focusedCommentId=16558739=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16558739],
>
> {quote}Class AvailableSpaceBlockPlacementPolicy extends
> BlockPlacementPolicyDefault. But it doesn't change the behavior of choosing
> the first node in BlockPlacementPolicyDefault, so even with this new feature,
> the local DN is always chosen as the first DN (of course when it is not
> excluded), and the new feature only changes the selection of the rest of the
> two DNs.
> {quote}
> I'm file this Jira as I groom Cloudera's internal Jira and found this
> unreported issue. We do have a customer hitting this problem. I don't have a
> fix, but thought it would be beneficial to report it to Apache Jira.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14578) AvailableSpaceBlockPlacementPolicy always prefers local node
[
https://issues.apache.org/jira/browse/HDFS-14578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-14578:
Status: Patch Available (was: Open)
> AvailableSpaceBlockPlacementPolicy always prefers local node
>
>
> Key: HDFS-14578
> URL: https://issues.apache.org/jira/browse/HDFS-14578
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
>Affects Versions: 3.0.0-alpha1, 2.7.4, 2.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14578-WIP-01.patch, HDFS-14758-01.patch
>
>
> It looks like AvailableSpaceBlockPlacementPolicy prefers local disk just like
> in the BlockPlacementPolicyDefault
>
> As Yongjun mentioned in
> [HDFS-8131|https://issues.apache.org/jira/browse/HDFS-8131?focusedCommentId=16558739=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16558739],
>
> {quote}Class AvailableSpaceBlockPlacementPolicy extends
> BlockPlacementPolicyDefault. But it doesn't change the behavior of choosing
> the first node in BlockPlacementPolicyDefault, so even with this new feature,
> the local DN is always chosen as the first DN (of course when it is not
> excluded), and the new feature only changes the selection of the rest of the
> two DNs.
> {quote}
> I'm file this Jira as I groom Cloudera's internal Jira and found this
> unreported issue. We do have a customer hitting this problem. I don't have a
> fix, but thought it would be beneficial to report it to Apache Jira.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15087) RBF: Balance/Rename across federation namespaces
[ https://issues.apache.org/jira/browse/HDFS-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009739#comment-17009739 ] Jinglun commented on HDFS-15087: Hi [~linyiqun], thanks your suggestion ! I prepared a initial patch based on xiaomi branch. There are few annotation in the patch. So I write some annotations below for the important classes. * FRHelper, FederationRenameV2, FederationRenameV1(Deprecated)：Serialization & Deserialization. * FSFedRenameV2Op: It handles saveSubTree and graftSubTree rpcs. Like FSDirRenameOp. * BasicInfo: It contains the basic information of an HFR job. * BlockLinker: It does all the hard links. Including collecting and hardlink. * BlocksToDup: Blocks in batch. * FsDatasetImpl: method addBlocksToNewPool is added to hard link replicas. * IDConsumer: It's used by NameNode. After the Ids are pre-allocated, we use IDConsumer to consume all the allocated Ids. * LockHelper: It helps release and restore all the write locks. * ParallelConsumer: The super class of ChecksumDirectoriesChecker. * ChecksumDirectoriesChecker: After the HardLink phase done, It verifies the source path and destination path. * JobScheduler, Job, JobContext: The Scheduler model in the design doc. * FederationRenameProcedure: It's the HFR. * DistCpProcedure: Use distcp instead of saveSubTree + graftSubTree + HardLink. > RBF: Balance/Rename across federation namespaces > > > Key: HDFS-15087 > URL: https://issues.apache.org/jira/browse/HDFS-15087 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Priority: Major > Attachments: HDFS-15087.initial.patch, HFR_Rename Across Federation > Namespaces.pdf > > > The Xiaomi storage team has developed a new feature called HFR(HDFS > Federation Rename) that enables us to do balance/rename across federation > namespaces. The idea is to first move the meta to the dst NameNode and then > link all the replicas. It has been working in our largest production cluster > for 2 months. We use it to balance the namespaces. It turns out HFR is fast > and flexible. The detail could be found in the design doc. > Looking forward to a lively discussion. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15087) RBF: Balance/Rename across federation namespaces
[ https://issues.apache.org/jira/browse/HDFS-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinglun updated HDFS-15087: --- Attachment: HDFS-15087.initial.patch > RBF: Balance/Rename across federation namespaces > > > Key: HDFS-15087 > URL: https://issues.apache.org/jira/browse/HDFS-15087 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Priority: Major > Attachments: HDFS-15087.initial.patch, HFR_Rename Across Federation > Namespaces.pdf > > > The Xiaomi storage team has developed a new feature called HFR(HDFS > Federation Rename) that enables us to do balance/rename across federation > namespaces. The idea is to first move the meta to the dst NameNode and then > link all the replicas. It has been working in our largest production cluster > for 2 months. We use it to balance the namespaces. It turns out HFR is fast > and flexible. The detail could be found in the design doc. > Looking forward to a lively discussion. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15096) RBF : GetServerDefaults Should be Cached At Router
[ https://issues.apache.org/jira/browse/HDFS-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15096: Status: Patch Available (was: Open) > RBF : GetServerDefaults Should be Cached At Router > -- > > Key: HDFS-15096 > URL: https://issues.apache.org/jira/browse/HDFS-15096 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15096-01.patch > > > Presently each getServerDefault calls are triggered to the Namespace, The > DFSClient caches the getServerDefaults(), similarly the Router can also cache > the getServerDefaults(), as router connects to multiple clients, this > improves performance for subsequent calls. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15096) RBF : GetServerDefaults Should be Cached At Router
[ https://issues.apache.org/jira/browse/HDFS-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15096: Attachment: HDFS-15096-01.patch > RBF : GetServerDefaults Should be Cached At Router > -- > > Key: HDFS-15096 > URL: https://issues.apache.org/jira/browse/HDFS-15096 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15096-01.patch > > > Presently each getServerDefault calls are triggered to the Namespace, The > DFSClient caches the getServerDefaults(), similarly the Router can also cache > the getServerDefaults(), as router connects to multiple clients, this > improves performance for subsequent calls. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15094) RBF : Reuse ugi string in ConnectionPoolID
[ https://issues.apache.org/jira/browse/HDFS-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009726#comment-17009726 ] Ayush Saxena commented on HDFS-15094: - Updated patch, to store ugi.toString() and reuse rather than recalculating for hash and equals methods. Did profiling for this, getConnection() prior used to take 1497 was reduced to 927. A different approach thought was to use ugi.hashcode() for hash and ugi.equals() in the equals methods, that in non secured environment reduced the time to ~750, but in case of secured environment, the Subject of ugi is changed at the router to set the user name, So in case of secured environment, each call would create a new connection, as equals method of ugi, compares Subject, which wont be same. So didn't use this approach. And limited to just storing the ugi.toString(). > RBF : Reuse ugi string in ConnectionPoolID > -- > > Key: HDFS-15094 > URL: https://issues.apache.org/jira/browse/HDFS-15094 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15094-01.patch > > > The connectionPoolID Hash Code and equals contains ugi.toString(), These > methods are used as part of getConnection() in ConnectionManager as part of > every call. > The ugi.toString() eats up considerable amount of time, the Hash Calculation > itself is ~10 percent of the total time of the call. And even more for the > equals method. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15094) RBF : Reuse ugi string in ConnectionPoolID
[ https://issues.apache.org/jira/browse/HDFS-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15094: Status: Patch Available (was: Open) > RBF : Reuse ugi string in ConnectionPoolID > -- > > Key: HDFS-15094 > URL: https://issues.apache.org/jira/browse/HDFS-15094 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15094-01.patch > > > The connectionPoolID Hash Code and equals contains ugi.toString(), These > methods are used as part of getConnection() in ConnectionManager as part of > every call. > The ugi.toString() eats up considerable amount of time, the Hash Calculation > itself is ~10 percent of the total time of the call. And even more for the > equals method. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15094) RBF : Reuse ugi string in ConnectionPoolID
[ https://issues.apache.org/jira/browse/HDFS-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15094: Attachment: HDFS-15094-01.patch > RBF : Reuse ugi string in ConnectionPoolID > -- > > Key: HDFS-15094 > URL: https://issues.apache.org/jira/browse/HDFS-15094 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15094-01.patch > > > The connectionPoolID Hash Code and equals contains ugi.toString(), These > methods are used as part of getConnection() in ConnectionManager as part of > every call. > The ugi.toString() eats up considerable amount of time, the Hash Calculation > itself is ~10 percent of the total time of the call. And even more for the > equals method. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14578) AvailableSpaceBlockPlacementPolicy always prefers local node
[
https://issues.apache.org/jira/browse/HDFS-14578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena reassigned HDFS-14578:
---
Assignee: Ayush Saxena
> AvailableSpaceBlockPlacementPolicy always prefers local node
>
>
> Key: HDFS-14578
> URL: https://issues.apache.org/jira/browse/HDFS-14578
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
>Affects Versions: 2.8.0, 2.7.4, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14578-WIP-01.patch
>
>
> It looks like AvailableSpaceBlockPlacementPolicy prefers local disk just like
> in the BlockPlacementPolicyDefault
>
> As Yongjun mentioned in
> [HDFS-8131|https://issues.apache.org/jira/browse/HDFS-8131?focusedCommentId=16558739=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16558739],
>
> {quote}Class AvailableSpaceBlockPlacementPolicy extends
> BlockPlacementPolicyDefault. But it doesn't change the behavior of choosing
> the first node in BlockPlacementPolicyDefault, so even with this new feature,
> the local DN is always chosen as the first DN (of course when it is not
> excluded), and the new feature only changes the selection of the rest of the
> two DNs.
> {quote}
> I'm file this Jira as I groom Cloudera's internal Jira and found this
> unreported issue. We do have a customer hitting this problem. I don't have a
> fix, but thought it would be beneficial to report it to Apache Jira.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14578) AvailableSpaceBlockPlacementPolicy always prefers local node
[
https://issues.apache.org/jira/browse/HDFS-14578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009576#comment-17009576
]
Vinayakumar B commented on HDFS-14578:
--
Idea looks good. You can now separate the test and keep default config to false
to keep existing behavior.
> AvailableSpaceBlockPlacementPolicy always prefers local node
>
>
> Key: HDFS-14578
> URL: https://issues.apache.org/jira/browse/HDFS-14578
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
>Affects Versions: 2.8.0, 2.7.4, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14578-WIP-01.patch
>
>
> It looks like AvailableSpaceBlockPlacementPolicy prefers local disk just like
> in the BlockPlacementPolicyDefault
>
> As Yongjun mentioned in
> [HDFS-8131|https://issues.apache.org/jira/browse/HDFS-8131?focusedCommentId=16558739=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16558739],
>
> {quote}Class AvailableSpaceBlockPlacementPolicy extends
> BlockPlacementPolicyDefault. But it doesn't change the behavior of choosing
> the first node in BlockPlacementPolicyDefault, so even with this new feature,
> the local DN is always chosen as the first DN (of course when it is not
> excluded), and the new feature only changes the selection of the rest of the
> two DNs.
> {quote}
> I'm file this Jira as I groom Cloudera's internal Jira and found this
> unreported issue. We do have a customer hitting this problem. I don't have a
> fix, but thought it would be beneficial to report it to Apache Jira.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15097) Purge log in KMS and HttpFS
[
https://issues.apache.org/jira/browse/HDFS-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009560#comment-17009560
]
Doris Gu commented on HDFS-15097:
-
[~jzhuge], [~xiaochen], please check, thanks in advance!
> Purge log in KMS and HttpFS
> ---
>
> Key: HDFS-15097
> URL: https://issues.apache.org/jira/browse/HDFS-15097
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: httpfs, kms
>Affects Versions: 3.0.3, 3.3.0, 3.2.1, 3.1.3
>Reporter: Doris Gu
>Assignee: Doris Gu
>Priority: Minor
> Attachments: HDFS-15097.001.patch
>
>
> KMS and HttpFS uses ConfigurationWithLogging instead of Configuration, which
> logs a configuration object each access. It's more like a development use.
> {code:java}
> 2020-01-07 16:52:00,456 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
> Got hadoop.security.instrumentation.requires.admin = 'false'
> 2020-01-07 16:52:00,456 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
> Got hadoop.security.instrumentation.requires.admin = 'false' (default
> 'false')
> 2020-01-07 16:52:15,091 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
> Got hadoop.security.instrumentation.requires.admin = 'false'
> 2020-01-07 16:52:15,091 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
> Got hadoop.security.instrumentation.requires.admin = 'false' (default 'false')
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15097) Purge log in KMS and HttpFS
[
https://issues.apache.org/jira/browse/HDFS-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doris Gu updated HDFS-15097:
Attachment: HDFS-15097.001.patch
> Purge log in KMS and HttpFS
> ---
>
> Key: HDFS-15097
> URL: https://issues.apache.org/jira/browse/HDFS-15097
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: httpfs, kms
>Affects Versions: 3.0.3, 3.3.0, 3.2.1, 3.1.3
>Reporter: Doris Gu
>Assignee: Doris Gu
>Priority: Minor
> Attachments: HDFS-15097.001.patch
>
>
> KMS and HttpFS uses ConfigurationWithLogging instead of Configuration, which
> logs a configuration object each access. It's more like a development use.
> {code:java}
> 2020-01-07 16:52:00,456 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
> Got hadoop.security.instrumentation.requires.admin = 'false'
> 2020-01-07 16:52:00,456 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
> Got hadoop.security.instrumentation.requires.admin = 'false' (default
> 'false')
> 2020-01-07 16:52:15,091 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
> Got hadoop.security.instrumentation.requires.admin = 'false'
> 2020-01-07 16:52:15,091 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
> Got hadoop.security.instrumentation.requires.admin = 'false' (default 'false')
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15097) Purge log in KMS and HttpFS
[
https://issues.apache.org/jira/browse/HDFS-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doris Gu updated HDFS-15097:
Description:
KMS and HttpFS uses ConfigurationWithLogging instead of Configuration, which
logs a configuration object each access. It's more like a development use.
{code:java}
// 2020-01-07 16:52:00,456 INFO
org.apache.hadoop.conf.ConfigurationWithLogging: Got
hadoop.security.instrumentation.requires.admin = 'false' 2020-01-07
16:52:00,456 INFO org.apache.hadoop.conf.ConfigurationWithLogging: Got
hadoop.security.instrumentation.requires.admin = 'false' (default 'false')
2020-01-07 16:52:15,091 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
Got hadoop.security.instrumentation.requires.admin = 'false' 2020-01-07
16:52:15,091 INFO org.apache.hadoop.conf.ConfigurationWithLogging: Got
hadoop.security.instrumentation.requires.admin = 'false' (default 'false')
{code}
> Purge log in KMS and HttpFS
> ---
>
> Key: HDFS-15097
> URL: https://issues.apache.org/jira/browse/HDFS-15097
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: httpfs, kms
>Affects Versions: 3.0.3, 3.3.0, 3.2.1, 3.1.3
>Reporter: Doris Gu
>Assignee: Doris Gu
>Priority: Minor
>
> KMS and HttpFS uses ConfigurationWithLogging instead of Configuration, which
> logs a configuration object each access. It's more like a development use.
> {code:java}
> // 2020-01-07 16:52:00,456 INFO
> org.apache.hadoop.conf.ConfigurationWithLogging: Got
> hadoop.security.instrumentation.requires.admin = 'false' 2020-01-07
> 16:52:00,456 INFO org.apache.hadoop.conf.ConfigurationWithLogging: Got
> hadoop.security.instrumentation.requires.admin = 'false' (default 'false')
> 2020-01-07 16:52:15,091 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
> Got hadoop.security.instrumentation.requires.admin = 'false' 2020-01-07
> 16:52:15,091 INFO org.apache.hadoop.conf.ConfigurationWithLogging: Got
> hadoop.security.instrumentation.requires.admin = 'false' (default 'false')
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15097) Purge log in KMS and HttpFS
[
https://issues.apache.org/jira/browse/HDFS-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doris Gu updated HDFS-15097:
Description:
KMS and HttpFS uses ConfigurationWithLogging instead of Configuration, which
logs a configuration object each access. It's more like a development use.
{code:java}
2020-01-07 16:52:00,456 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
Got hadoop.security.instrumentation.requires.admin = 'false'
2020-01-07 16:52:00,456 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
Got hadoop.security.instrumentation.requires.admin = 'false' (default 'false')
2020-01-07 16:52:15,091 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
Got hadoop.security.instrumentation.requires.admin = 'false'
2020-01-07 16:52:15,091 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
Got hadoop.security.instrumentation.requires.admin = 'false' (default 'false')
{code}
was:
KMS and HttpFS uses ConfigurationWithLogging instead of Configuration, which
logs a configuration object each access. It's more like a development use.
{code:java}
// 2020-01-07 16:52:00,456 INFO
org.apache.hadoop.conf.ConfigurationWithLogging: Got
hadoop.security.instrumentation.requires.admin = 'false' 2020-01-07
16:52:00,456 INFO org.apache.hadoop.conf.ConfigurationWithLogging: Got
hadoop.security.instrumentation.requires.admin = 'false' (default 'false')
2020-01-07 16:52:15,091 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
Got hadoop.security.instrumentation.requires.admin = 'false' 2020-01-07
16:52:15,091 INFO org.apache.hadoop.conf.ConfigurationWithLogging: Got
hadoop.security.instrumentation.requires.admin = 'false' (default 'false')
{code}
> Purge log in KMS and HttpFS
> ---
>
> Key: HDFS-15097
> URL: https://issues.apache.org/jira/browse/HDFS-15097
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: httpfs, kms
>Affects Versions: 3.0.3, 3.3.0, 3.2.1, 3.1.3
>Reporter: Doris Gu
>Assignee: Doris Gu
>Priority: Minor
>
> KMS and HttpFS uses ConfigurationWithLogging instead of Configuration, which
> logs a configuration object each access. It's more like a development use.
> {code:java}
> 2020-01-07 16:52:00,456 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
> Got hadoop.security.instrumentation.requires.admin = 'false'
> 2020-01-07 16:52:00,456 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
> Got hadoop.security.instrumentation.requires.admin = 'false' (default
> 'false')
> 2020-01-07 16:52:15,091 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
> Got hadoop.security.instrumentation.requires.admin = 'false'
> 2020-01-07 16:52:15,091 INFO org.apache.hadoop.conf.ConfigurationWithLogging:
> Got hadoop.security.instrumentation.requires.admin = 'false' (default 'false')
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15097) Purge log in KMS and HttpFS
Doris Gu created HDFS-15097: --- Summary: Purge log in KMS and HttpFS Key: HDFS-15097 URL: https://issues.apache.org/jira/browse/HDFS-15097 Project: Hadoop HDFS Issue Type: Improvement Components: httpfs, kms Affects Versions: 3.1.3, 3.2.1, 3.0.3, 3.3.0 Reporter: Doris Gu Assignee: Doris Gu -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
