[jira] [Commented] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177481#comment-17177481 ] Hemanth Boyina commented on HDFS-15530: --- +1 > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15530.patch > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15530) Wrong word in the HDFS Router configuration file
[
https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177457#comment-17177457
]
Hadoop QA commented on HDFS-15530:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
14s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 40s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
24s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
[jira] [Commented] (HDFS-15290) NPE in HttpServer during NameNode startup
[
https://issues.apache.org/jira/browse/HDFS-15290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177456#comment-17177456
]
Hadoop QA commented on HDFS-15290:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
0s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 66 unchanged - 0 fixed = 67 total (was 66) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 43s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
[jira] [Updated] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sha Fanghao updated HDFS-15530: --- Attachment: HDFS-15530.patch Status: Patch Available (was: Reopened) > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15530.patch > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sha Fanghao updated HDFS-15530: --- Attachment: (was: HDFS-15530.patch) > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15530.patch > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177415#comment-17177415 ] Sha Fanghao edited comment on HDFS-15530 at 8/14/20, 2:11 AM: -- [~elgoiri] Sorry，it's a wrong click to “resolved”，and now I reopen it again. was (Author: lausaa): sorry，wrong click to “resolved”，and now reopen it again. > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15530.patch > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sha Fanghao reopened HDFS-15530: sorry，wrong click to “resolved”，and now reopen it again. > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15530.patch > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15290) NPE in HttpServer during NameNode startup
[ https://issues.apache.org/jira/browse/HDFS-15290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simbarashe Dzinamarira updated HDFS-15290: -- Attachment: HDFS-15290.001.patch Target Version/s: 2.10.0 (was: 2.10.1) Affects Version/s: 2.10.0 3.3.0 Status: Patch Available (was: Open) > NPE in HttpServer during NameNode startup > - > > Key: HDFS-15290 > URL: https://issues.apache.org/jira/browse/HDFS-15290 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.3.0, 2.10.0, 2.7.8 >Reporter: Konstantin Shvachko >Assignee: Simbarashe Dzinamarira >Priority: Major > Attachments: HDFS-15290.001.patch > > > When NameNode starts it first starts HttpServer, then starts loading fsImage > and edits. While loading the namesystem field in NameNode is null. I saw that > a StandbyNode sends a checkpoint request, which fails with NPE because > NNStorage is not instantiated yet. > We should check the NameNode startup status before accepting checkpoint > requests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15290) NPE in HttpServer during NameNode startup
[ https://issues.apache.org/jira/browse/HDFS-15290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simbarashe Dzinamarira updated HDFS-15290: -- Attachment: (was: HDFS-15290.001.patch) > NPE in HttpServer during NameNode startup > - > > Key: HDFS-15290 > URL: https://issues.apache.org/jira/browse/HDFS-15290 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.8 >Reporter: Konstantin Shvachko >Assignee: Simbarashe Dzinamarira >Priority: Major > > When NameNode starts it first starts HttpServer, then starts loading fsImage > and edits. While loading the namesystem field in NameNode is null. I saw that > a StandbyNode sends a checkpoint request, which fails with NPE because > NNStorage is not instantiated yet. > We should check the NameNode startup status before accepting checkpoint > requests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15533) Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside
[
https://issues.apache.org/jira/browse/HDFS-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15533:
---
Summary: Provide DFS API compatible class, but use
ViewFileSystemOverloadScheme inside (was: Provide DFS API compatible call, but
use ViewFileSystemOverloadScheme inside)
> Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside
> -
>
> Key: HDFS-15533
> URL: https://issues.apache.org/jira/browse/HDFS-15533
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: dfs, viewfs
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> I have been working on a thought from last week is that, we wanted to provide
> DFS compatible APIs with mount functionality. So, that existing DFS
> applications can work with out class cast issues.
> When we tested with other components like Hive and HBase, I noticed some
> classcast issues.
> {code:java}
> HBase example:
> java.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystem at
> org.apache.hadoop.hbase.util.FSUtils.getDFSHedgedReadMetrics(FSUtils.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl.(MetricsRegionServerWrapperImpl.java:146)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1594)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1001)
> at java.lang.Thread.run(Thread.java:748){code}
> {code:java}
> Hive:
> |io.AcidUtils|: Failed to get files with ID; using regular API: Only
> supported for DFS; got class
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme{code}
> SO, here the implementation details are like follows:
> We extended DistributedFileSystem and created a class called "
> ViewDistributedFileSystem"
> This vfs=ViewFirstibutedFileSystem, try to initialize
> ViewFileSystemOverloadScheme. If success call will delegate to vfs. If fails
> to initialize due to no mount points, or other errors, it will just fallback
> to regular DFS init. If users does not configure any mount, system will
> behave exactly like today's DFS. If there are mount points, vfs functionality
> will come under DFS.
> I will a patch and will post it in some time.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15533) Provide DFS API compatible call, but use ViewFileSystemOverloadScheme inside
Uma Maheswara Rao G created HDFS-15533:
--
Summary: Provide DFS API compatible call, but use
ViewFileSystemOverloadScheme inside
Key: HDFS-15533
URL: https://issues.apache.org/jira/browse/HDFS-15533
Project: Hadoop HDFS
Issue Type: Sub-task
Components: dfs, viewfs
Affects Versions: 3.4.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
I have been working on a thought from last week is that, we wanted to provide
DFS compatible APIs with mount functionality. So, that existing DFS
applications can work with out class cast issues.
When we tested with other components like Hive and HBase, I noticed some
classcast issues.
{code:java}
HBase example:
java.lang.ClassCastException:
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
org.apache.hadoop.hdfs.DistributedFileSystem at
org.apache.hadoop.hbase.util.FSUtils.getDFSHedgedReadMetrics(FSUtils.java:1748)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl.(MetricsRegionServerWrapperImpl.java:146)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1594)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1001)
at java.lang.Thread.run(Thread.java:748){code}
{code:java}
Hive:
|io.AcidUtils|: Failed to get files with ID; using regular API: Only supported
for DFS; got class
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme{code}
SO, here the implementation details are like follows:
We extended DistributedFileSystem and created a class called "
ViewDistributedFileSystem"
This vfs=ViewFirstibutedFileSystem, try to initialize
ViewFileSystemOverloadScheme. If success call will delegate to vfs. If fails
to initialize due to no mount points, or other errors, it will just fallback to
regular DFS init. If users does not configure any mount, system will behave
exactly like today's DFS. If there are mount points, vfs functionality will
come under DFS.
I will a patch and will post it in some time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15329) Provide FileContext based ViewFSOverloadScheme implementation
[ https://issues.apache.org/jira/browse/HDFS-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177370#comment-17177370 ] Uma Maheswara Rao G commented on HDFS-15329: Thanks [~abhishekd] I was trying to assign this issue to you, but somehow I am seeing ur name getting listed there. If you are getting "Assign to me" option, feel free to assign it urself. > Provide FileContext based ViewFSOverloadScheme implementation > - > > Key: HDFS-15329 > URL: https://issues.apache.org/jira/browse/HDFS-15329 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Priority: Major > > This Jira to track for FileContext based ViewFSOverloadScheme implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15329) Provide FileContext based ViewFSOverloadScheme implementation
[ https://issues.apache.org/jira/browse/HDFS-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDFS-15329: -- Assignee: (was: Uma Maheswara Rao G) > Provide FileContext based ViewFSOverloadScheme implementation > - > > Key: HDFS-15329 > URL: https://issues.apache.org/jira/browse/HDFS-15329 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Priority: Major > > This Jira to track for FileContext based ViewFSOverloadScheme implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15329) Provide FileContext based ViewFSOverloadScheme implementation
[ https://issues.apache.org/jira/browse/HDFS-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDFS-15329: -- Assignee: Uma Maheswara Rao G > Provide FileContext based ViewFSOverloadScheme implementation > - > > Key: HDFS-15329 > URL: https://issues.apache.org/jira/browse/HDFS-15329 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > This Jira to track for FileContext based ViewFSOverloadScheme implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15329) Provide FileContext based ViewFSOverloadScheme implementation
[ https://issues.apache.org/jira/browse/HDFS-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDFS-15329: -- Assignee: (was: Uma Maheswara Rao G) > Provide FileContext based ViewFSOverloadScheme implementation > - > > Key: HDFS-15329 > URL: https://issues.apache.org/jira/browse/HDFS-15329 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Priority: Major > > This Jira to track for FileContext based ViewFSOverloadScheme implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15532) listFiles on root will fail if fallback root has file
Uma Maheswara Rao G created HDFS-15532:
--
Summary: listFiles on root will fail if fallback root has file
Key: HDFS-15532
URL: https://issues.apache.org/jira/browse/HDFS-15532
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
listFiles implementation gets the RemoteIterator created in InternalViewFSDirFs
as the root is an InternalViewFSDir.
If there is a fallback and a file exist at root level, it would have collected
when collecting locatedStatuses.
When its iterating over to that fallbacks file from RemoteIterator (which was
returned from InternalViewFSDirFs ), iterator's next will will call
getFileBlockLocations if it's a file.
{code:java}
@Override
public LocatedFileStatus next() throws IOException {
System.out.println(this);
if (!hasNext()) {
throw new NoSuchElementException("No more entries in " + f);
}
FileStatus result = stats[i++];
// for files, use getBlockLocations(FileStatus, int, int) to avoid
// calling getFileStatus(Path) to load the FileStatus again
BlockLocation[] locs = result.isFile() ?
getFileBlockLocations(result, 0, result.getLen()) :
null;
return new LocatedFileStatus(result, locs);
}{code}
this getFileBlockLocations will be made on InternalViewFSDirFs, as that
Iterator created originally from that fs.
InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases. It's
always expecting "/", this means it always assuming the dir.
But with the fallback and returning Iterator from InternalViewFSDirFs, will
create problems.
Probably we need to handle fallback case in getFileBlockLocations as well.(
Fallback only should be the reason for call coming to InternalViewFSDirFs with
other than "/")
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15290) NPE in HttpServer during NameNode startup
[ https://issues.apache.org/jira/browse/HDFS-15290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simbarashe Dzinamarira updated HDFS-15290: -- Attachment: HDFS-15290.001.patch > NPE in HttpServer during NameNode startup > - > > Key: HDFS-15290 > URL: https://issues.apache.org/jira/browse/HDFS-15290 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.8 >Reporter: Konstantin Shvachko >Assignee: Simbarashe Dzinamarira >Priority: Major > Attachments: HDFS-15290.001.patch > > > When NameNode starts it first starts HttpServer, then starts loading fsImage > and edits. While loading the namesystem field in NameNode is null. I saw that > a StandbyNode sends a checkpoint request, which fails with NPE because > NNStorage is not instantiated yet. > We should check the NameNode startup status before accepting checkpoint > requests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15290) NPE in HttpServer during NameNode startup
[ https://issues.apache.org/jira/browse/HDFS-15290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simbarashe Dzinamarira reassigned HDFS-15290: - Assignee: Simbarashe Dzinamarira > NPE in HttpServer during NameNode startup > - > > Key: HDFS-15290 > URL: https://issues.apache.org/jira/browse/HDFS-15290 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.8 >Reporter: Konstantin Shvachko >Assignee: Simbarashe Dzinamarira >Priority: Major > > When NameNode starts it first starts HttpServer, then starts loading fsImage > and edits. While loading the namesystem field in NameNode is null. I saw that > a StandbyNode sends a checkpoint request, which fails with NPE because > NNStorage is not instantiated yet. > We should check the NameNode startup status before accepting checkpoint > requests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-15531) Namenode UI: List snapshots in separate table for each snapshottable directory
[ https://issues.apache.org/jira/browse/HDFS-15531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-15531 started by Vivek Ratnavel Subramanian. - > Namenode UI: List snapshots in separate table for each snapshottable directory > -- > > Key: HDFS-15531 > URL: https://issues.apache.org/jira/browse/HDFS-15531 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ui >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15531) Namenode UI: List snapshots in separate table for each snapshottable directory
Vivek Ratnavel Subramanian created HDFS-15531: - Summary: Namenode UI: List snapshots in separate table for each snapshottable directory Key: HDFS-15531 URL: https://issues.apache.org/jira/browse/HDFS-15531 Project: Hadoop HDFS Issue Type: Sub-task Components: ui Reporter: Vivek Ratnavel Subramanian Assignee: Vivek Ratnavel Subramanian -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177216#comment-17177216 ] Íñigo Goiri commented on HDFS-15530: [~lausaa], why did you mark it as fixed? The error is there. > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15530.patch > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15496) Add UI for deleted snapshots
[ https://issues.apache.org/jira/browse/HDFS-15496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-15496: --- Fix Version/s: 3.4.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Add UI for deleted snapshots > > > Key: HDFS-15496 > URL: https://issues.apache.org/jira/browse/HDFS-15496 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Mukul Kumar Singh >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Fix For: 3.4.0 > > > Add UI for deleted snapshots > a) Show the list of snapshots per snapshottable directory > b) Add deleted status in the JMX output for the Snapshot along with a snap ID > e) NN UI, should sort the snapshots for snapIds. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sha Fanghao updated HDFS-15530: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15530.patch > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sha Fanghao updated HDFS-15530: --- Priority: Minor (was: Major) > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15530.patch > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sha Fanghao updated HDFS-15530: --- Attachment: HDFS-15530.patch Status: Patch Available (was: Open) > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Priority: Major > Fix For: 3.4.0 > > Attachments: HDFS-15530.patch > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sha Fanghao updated HDFS-15530: --- Attachment: (was: HDFS-15530.patch) > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Priority: Major > Fix For: 3.4.0 > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sha Fanghao updated HDFS-15530: --- Attachment: (was: HDFS-15530.patch) > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Priority: Major > Fix For: 3.4.0 > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sha Fanghao updated HDFS-15530: --- Status: Open (was: Patch Available) > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Priority: Major > Fix For: 3.4.0 > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sha Fanghao updated HDFS-15530: --- Attachment: HDFS-15530.patch Status: Patch Available (was: Reopened) > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Priority: Major > Fix For: 3.4.0 > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sha Fanghao updated HDFS-15530: --- Attachment: HDFS-15530.patch > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Priority: Major > Fix For: 3.4.0 > > Attachments: HDFS-15530.patch > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sha Fanghao updated HDFS-15530: --- Description: UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. (was: UPDATE was misspelled as UPDATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL.) > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Priority: Major > Fix For: 3.4.0 > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sha Fanghao reopened HDFS-15530: > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Sha Fanghao >Priority: Major > Fix For: 3.4.0 > > > UPDATE was misspelled as UPDATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sha Fanghao updated HDFS-15530: --- Issue Type: Improvement (was: Task) > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Priority: Major > Fix For: 3.4.0 > > > UPDATE was misspelled as UPDATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15530) Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sha Fanghao resolved HDFS-15530. Resolution: Fixed > Wrong word in the HDFS Router configuration file > > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Sha Fanghao >Priority: Major > Fix For: 3.4.0 > > > UPDATE was misspelled as UPDATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15530) Wrong word in the HDFS Router configuration file
Sha Fanghao created HDFS-15530: -- Summary: Wrong word in the HDFS Router configuration file Key: HDFS-15530 URL: https://issues.apache.org/jira/browse/HDFS-15530 Project: Hadoop HDFS Issue Type: Task Reporter: Sha Fanghao Fix For: 3.4.0 UPDATE was misspelled as UPDATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15515) mkdirs on fallback should throw IOE out instead of suppressing and returning false
[
https://issues.apache.org/jira/browse/HDFS-15515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177012#comment-17177012
]
Steve Loughran commented on HDFS-15515:
---
cherry-picked to branch-3.3, so people get their stack traces faster :)
> mkdirs on fallback should throw IOE out instead of suppressing and returning
> false
> --
>
> Key: HDFS-15515
> URL: https://issues.apache.org/jira/browse/HDFS-15515
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Fix For: 3.3.1
>
>
> Currently when doing mkdirs on fallback dir, we catching IOE and returning
> false.
> I think we should just throw IOE out as the fs#mkdirs throws IOE out.
> I noticed a case when we attempt to create .reserved dirs, NN throws
> HadoopIAE.
> But we will catch and return false. Here exception should be thrown out.
> {code:java}
> try {
> return linkedFallbackFs.mkdirs(dirToCreate, permission);
> } catch (IOException e) {
> if (LOG.isDebugEnabled()) {
> StringBuilder msg =
> new StringBuilder("Failed to create ").append(dirToCreate)
> .append(" at fallback : ")
> .append(linkedFallbackFs.getUri());
> LOG.debug(msg.toString(), e);
> }
> return false;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15515) mkdirs on fallback should throw IOE out instead of suppressing and returning false
[
https://issues.apache.org/jira/browse/HDFS-15515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-15515:
--
Fix Version/s: (was: 3.4.0)
3.3.1
> mkdirs on fallback should throw IOE out instead of suppressing and returning
> false
> --
>
> Key: HDFS-15515
> URL: https://issues.apache.org/jira/browse/HDFS-15515
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Fix For: 3.3.1
>
>
> Currently when doing mkdirs on fallback dir, we catching IOE and returning
> false.
> I think we should just throw IOE out as the fs#mkdirs throws IOE out.
> I noticed a case when we attempt to create .reserved dirs, NN throws
> HadoopIAE.
> But we will catch and return false. Here exception should be thrown out.
> {code:java}
> try {
> return linkedFallbackFs.mkdirs(dirToCreate, permission);
> } catch (IOException e) {
> if (LOG.isDebugEnabled()) {
> StringBuilder msg =
> new StringBuilder("Failed to create ").append(dirToCreate)
> .append(" at fallback : ")
> .append(linkedFallbackFs.getUri());
> LOG.debug(msg.toString(), e);
> }
> return false;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15039) Cache meta file length of FinalizedReplica to reduce call File.length()
[
https://issues.apache.org/jira/browse/HDFS-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176885#comment-17176885
]
lindongdong commented on HDFS-15039:
OK, thanks~~
> Cache meta file length of FinalizedReplica to reduce call File.length()
> ---
>
> Key: HDFS-15039
> URL: https://issues.apache.org/jira/browse/HDFS-15039
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Yang Yun
>Assignee: Yang Yun
>Priority: Minor
> Attachments: HDFS-15039.006.patch, HDFS-15039.patch,
> HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch
>
>
> When use ReplicaCachingGetSpaceUsed to get the volume space used. It will
> call File.length() for every meta file of replica. That add more disk IO, we
> found the slow log as below. For finalized replica, the size of meta file is
> not changed, i think we can cache the value.
> {code:java}
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed:
> Refresh dfs used, bpid: BP-898717543-10.75.1.240-1519386995727 replicas
> size: 1166 dfsUsed: 72227113183 on volume:
> DS-3add8d62-d69a-4f5a-a29f-b7bbb400af2e duration: 17206ms{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14535) The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read
[ https://issues.apache.org/jira/browse/HDFS-14535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lujie reassigned HDFS-14535: Assignee: Zheng Hu > The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is > causing lots of heap allocation in HBase when using short-circut read > -- > > Key: HDFS-14535 > URL: https://issues.apache.org/jira/browse/HDFS-14535 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3 > > Attachments: HDFS-14535.patch > > > Our HBase team are trying to read the blocks from HDFS into pooled offheap > ByteBuffers directly (HBASE-21879), and recently we had some benchmark, > found that almost 45% heap allocation from the DFS client. The heap > allocation flame graph can be see here: > https://issues.apache.org/jira/secure/attachment/12970295/async-prof-pid-25042-alloc-2.svg > After checking the code path, we found that when requesting file descriptors > from a DomainPeer, we allocated huge 8KB buffer for BufferedOutputStream, > though the protocal content was quite small and just few bytes. > It made a heavy GC pressure for HBase when cacheHitRatio < 60%, which > increased the HBase P999 latency. Actually, we can pre-allocate a small > buffer for the BufferedOutputStream, such as 512 bytes, it's enough to read > the short-circuit fd protocal content. we've created a patch like that, and > the allocation flame graph show that after the patch, the heap allocation > from DFS client dropped from 45% to 27%, that's a very good thing I think. > see: > https://issues.apache.org/jira/secure/attachment/12970475/async-prof-pid-24534-alloc-2.svg > Hope this attached patch can be merged into HDFS trunk, also Hadoop-2.8.x, > HBase will benifit a lot from this. > Thanks. > For more details, can see here: > https://issues.apache.org/jira/browse/HBASE-22387?focusedCommentId=16851639=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16851639 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14535) The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read
[ https://issues.apache.org/jira/browse/HDFS-14535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lujie reassigned HDFS-14535: Assignee: (was: lujie) > The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is > causing lots of heap allocation in HBase when using short-circut read > -- > > Key: HDFS-14535 > URL: https://issues.apache.org/jira/browse/HDFS-14535 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Zheng Hu >Priority: Major > Fix For: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3 > > Attachments: HDFS-14535.patch > > > Our HBase team are trying to read the blocks from HDFS into pooled offheap > ByteBuffers directly (HBASE-21879), and recently we had some benchmark, > found that almost 45% heap allocation from the DFS client. The heap > allocation flame graph can be see here: > https://issues.apache.org/jira/secure/attachment/12970295/async-prof-pid-25042-alloc-2.svg > After checking the code path, we found that when requesting file descriptors > from a DomainPeer, we allocated huge 8KB buffer for BufferedOutputStream, > though the protocal content was quite small and just few bytes. > It made a heavy GC pressure for HBase when cacheHitRatio < 60%, which > increased the HBase P999 latency. Actually, we can pre-allocate a small > buffer for the BufferedOutputStream, such as 512 bytes, it's enough to read > the short-circuit fd protocal content. we've created a patch like that, and > the allocation flame graph show that after the patch, the heap allocation > from DFS client dropped from 45% to 27%, that's a very good thing I think. > see: > https://issues.apache.org/jira/secure/attachment/12970475/async-prof-pid-24534-alloc-2.svg > Hope this attached patch can be merged into HDFS trunk, also Hadoop-2.8.x, > HBase will benifit a lot from this. > Thanks. > For more details, can see here: > https://issues.apache.org/jira/browse/HBASE-22387?focusedCommentId=16851639=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16851639 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14535) The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read
[ https://issues.apache.org/jira/browse/HDFS-14535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lujie reassigned HDFS-14535: Assignee: lujie (was: Zheng Hu) > The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is > causing lots of heap allocation in HBase when using short-circut read > -- > > Key: HDFS-14535 > URL: https://issues.apache.org/jira/browse/HDFS-14535 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Zheng Hu >Assignee: lujie >Priority: Major > Fix For: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3 > > Attachments: HDFS-14535.patch > > > Our HBase team are trying to read the blocks from HDFS into pooled offheap > ByteBuffers directly (HBASE-21879), and recently we had some benchmark, > found that almost 45% heap allocation from the DFS client. The heap > allocation flame graph can be see here: > https://issues.apache.org/jira/secure/attachment/12970295/async-prof-pid-25042-alloc-2.svg > After checking the code path, we found that when requesting file descriptors > from a DomainPeer, we allocated huge 8KB buffer for BufferedOutputStream, > though the protocal content was quite small and just few bytes. > It made a heavy GC pressure for HBase when cacheHitRatio < 60%, which > increased the HBase P999 latency. Actually, we can pre-allocate a small > buffer for the BufferedOutputStream, such as 512 bytes, it's enough to read > the short-circuit fd protocal content. we've created a patch like that, and > the allocation flame graph show that after the patch, the heap allocation > from DFS client dropped from 45% to 27%, that's a very good thing I think. > see: > https://issues.apache.org/jira/secure/attachment/12970475/async-prof-pid-24534-alloc-2.svg > Hope this attached patch can be merged into HDFS trunk, also Hadoop-2.8.x, > HBase will benifit a lot from this. > Thanks. > For more details, can see here: > https://issues.apache.org/jira/browse/HBASE-22387?focusedCommentId=16851639=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16851639 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15329) Provide FileContext based ViewFSOverloadScheme implementation
[ https://issues.apache.org/jira/browse/HDFS-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176861#comment-17176861 ] Abhishek Das commented on HDFS-15329: - I have raised the PR for this [https://github.com/apache/hadoop/pull/2225] [~umamaheswararao] Please take a look at the implementation when you get a chance. > Provide FileContext based ViewFSOverloadScheme implementation > - > > Key: HDFS-15329 > URL: https://issues.apache.org/jira/browse/HDFS-15329 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > This Jira to track for FileContext based ViewFSOverloadScheme implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15329) Provide FileContext based ViewFSOverloadScheme implementation
[ https://issues.apache.org/jira/browse/HDFS-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176860#comment-17176860 ] Abhishek Das commented on HDFS-15329: - Sorry I also missed this comment. We have out custom FileSystem class derived from ViewFileSystemOverloadScheme. We have started testing internally with our other components. > Provide FileContext based ViewFSOverloadScheme implementation > - > > Key: HDFS-15329 > URL: https://issues.apache.org/jira/browse/HDFS-15329 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > This Jira to track for FileContext based ViewFSOverloadScheme implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15039) Cache meta file length of FinalizedReplica to reduce call File.length()
[
https://issues.apache.org/jira/browse/HDFS-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176858#comment-17176858
]
Yang Yun commented on HDFS-15039:
-
Thanks [~lindongdong] for the doubt.
Yes，the int type is for saving the memory. I don't think the covert has
perfomanced issue.
> Cache meta file length of FinalizedReplica to reduce call File.length()
> ---
>
> Key: HDFS-15039
> URL: https://issues.apache.org/jira/browse/HDFS-15039
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Yang Yun
>Assignee: Yang Yun
>Priority: Minor
> Attachments: HDFS-15039.006.patch, HDFS-15039.patch,
> HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch
>
>
> When use ReplicaCachingGetSpaceUsed to get the volume space used. It will
> call File.length() for every meta file of replica. That add more disk IO, we
> found the slow log as below. For finalized replica, the size of meta file is
> not changed, i think we can cache the value.
> {code:java}
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed:
> Refresh dfs used, bpid: BP-898717543-10.75.1.240-1519386995727 replicas
> size: 1166 dfsUsed: 72227113183 on volume:
> DS-3add8d62-d69a-4f5a-a29f-b7bbb400af2e duration: 17206ms{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15039) Cache meta file length of FinalizedReplica to reduce call File.length()
[
https://issues.apache.org/jira/browse/HDFS-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176827#comment-17176827
]
lindongdong commented on HDFS-15039:
[~hadoop_yangyun], for the last patch, I have one doubt:
why the metaLength is int type? for saving the memory? but when get it, it
return long value, there is a convert. is it bad to performance?
> Cache meta file length of FinalizedReplica to reduce call File.length()
> ---
>
> Key: HDFS-15039
> URL: https://issues.apache.org/jira/browse/HDFS-15039
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Yang Yun
>Assignee: Yang Yun
>Priority: Minor
> Attachments: HDFS-15039.006.patch, HDFS-15039.patch,
> HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch
>
>
> When use ReplicaCachingGetSpaceUsed to get the volume space used. It will
> call File.length() for every meta file of replica. That add more disk IO, we
> found the slow log as below. For finalized replica, the size of meta file is
> not changed, i think we can cache the value.
> {code:java}
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed:
> Refresh dfs used, bpid: BP-898717543-10.75.1.240-1519386995727 replicas
> size: 1166 dfsUsed: 72227113183 on volume:
> DS-3add8d62-d69a-4f5a-a29f-b7bbb400af2e duration: 17206ms{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
