[jira] [Comment Edited] (HDFS-16009) HDFS tiered storage support
[ https://issues.apache.org/jira/browse/HDFS-16009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340575#comment-17340575 ] JiangHua Zhu edited comment on HDFS-16009 at 5/7/21, 5:38 AM: -- Hi [~LeonG]. I noticed that [HDFS-15547|https://issues.apache.org/jira/browse/HDFS-15547] and [HDFS-15683|https://issues.apache.org/jira/browse/HDFS-15683] has solved a similar problem, which is a good feature. was (Author: jianghuazhu): Hi [~LeonG]. I noticed that [HDFS-15547|https://issues.apache.org/jira/browse/HDFS-15547] and [HDFS-15683|https://issues.apache.org/jira/browse/HDFS-15683] has solved a similar problem, which is a good feature. > HDFS tiered storage support > --- > > Key: HDFS-16009 > URL: https://issues.apache.org/jira/browse/HDFS-16009 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Priority: Major > > HDFS currently stores data blocks on different data nodes. These data nodes > are random within the scope of the network topology (local rack/different > racks/node group). > When the NameNode allocates DataNodes to the Block, it is randomly selected, > that is, each DataNode is selected equally. > We should make a hierarchical strategy. For example, DataNode1, DataNode2, > and DataNode3 are the same level, and DataNode4 and DataNode5 are the same > level. Because the hardware between DataNodes may be different, in this case, > it is necessary for us to do tiered storage, so that DataNodes with poor > hardware can store up to 50%, and 40% or even lower. This is conducive to > cluster maintenance and stability. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-16009) HDFS tiered storage support
[ https://issues.apache.org/jira/browse/HDFS-16009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340575#comment-17340575 ] JiangHua Zhu edited comment on HDFS-16009 at 5/7/21, 5:37 AM: -- Hi [~LeonG]. I noticed that [HDFS-15547|https://issues.apache.org/jira/browse/HDFS-15547] and [HDFS-15683|https://issues.apache.org/jira/browse/HDFS-15683] has solved a similar problem, which is a good feature. was (Author: jianghuazhu): Hi [~LeonG]. I noticed that [HDFS-15547|https://issues.apache.org/jira/browse/HDFS-15547] has solved a similar problem, which is a good feature. > HDFS tiered storage support > --- > > Key: HDFS-16009 > URL: https://issues.apache.org/jira/browse/HDFS-16009 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Priority: Major > > HDFS currently stores data blocks on different data nodes. These data nodes > are random within the scope of the network topology (local rack/different > racks/node group). > When the NameNode allocates DataNodes to the Block, it is randomly selected, > that is, each DataNode is selected equally. > We should make a hierarchical strategy. For example, DataNode1, DataNode2, > and DataNode3 are the same level, and DataNode4 and DataNode5 are the same > level. Because the hardware between DataNodes may be different, in this case, > it is necessary for us to do tiered storage, so that DataNodes with poor > hardware can store up to 50%, and 40% or even lower. This is conducive to > cluster maintenance and stability. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-16009) HDFS tiered storage support
[ https://issues.apache.org/jira/browse/HDFS-16009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340575#comment-17340575 ] JiangHua Zhu edited comment on HDFS-16009 at 5/7/21, 5:33 AM: -- Hi [~LeonG]. I noticed that [HDFS-15547|https://issues.apache.org/jira/browse/HDFS-15547] has solved a similar problem, which is a good feature. was (Author: jianghuazhu): Hi Leon Gao. I noticed that [HDFS-15547|https://issues.apache.org/jira/browse/HDFS-15547] has solved a similar problem, which is a good feature. > HDFS tiered storage support > --- > > Key: HDFS-16009 > URL: https://issues.apache.org/jira/browse/HDFS-16009 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Priority: Major > > HDFS currently stores data blocks on different data nodes. These data nodes > are random within the scope of the network topology (local rack/different > racks/node group). > When the NameNode allocates DataNodes to the Block, it is randomly selected, > that is, each DataNode is selected equally. > We should make a hierarchical strategy. For example, DataNode1, DataNode2, > and DataNode3 are the same level, and DataNode4 and DataNode5 are the same > level. Because the hardware between DataNodes may be different, in this case, > it is necessary for us to do tiered storage, so that DataNodes with poor > hardware can store up to 50%, and 40% or even lower. This is conducive to > cluster maintenance and stability. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-16009) HDFS tiered storage support
[ https://issues.apache.org/jira/browse/HDFS-16009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340575#comment-17340575 ] JiangHua Zhu edited comment on HDFS-16009 at 5/7/21, 5:33 AM: -- Hi Leon Gao. I noticed that [HDFS-15547|https://issues.apache.org/jira/browse/HDFS-15547] has solved a similar problem, which is a good feature. was (Author: jianghuazhu): Hi Leon Gao. I noticed that [HDFS-15547|https://issues.apache.org/jira/browse/HDFS-15547] [HDFS-15547] has solved a similar problem, which is a good feature. > HDFS tiered storage support > --- > > Key: HDFS-16009 > URL: https://issues.apache.org/jira/browse/HDFS-16009 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Priority: Major > > HDFS currently stores data blocks on different data nodes. These data nodes > are random within the scope of the network topology (local rack/different > racks/node group). > When the NameNode allocates DataNodes to the Block, it is randomly selected, > that is, each DataNode is selected equally. > We should make a hierarchical strategy. For example, DataNode1, DataNode2, > and DataNode3 are the same level, and DataNode4 and DataNode5 are the same > level. Because the hardware between DataNodes may be different, in this case, > it is necessary for us to do tiered storage, so that DataNodes with poor > hardware can store up to 50%, and 40% or even lower. This is conducive to > cluster maintenance and stability. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16009) HDFS tiered storage support
[ https://issues.apache.org/jira/browse/HDFS-16009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340575#comment-17340575 ] JiangHua Zhu commented on HDFS-16009: - Hi Leon Gao. I noticed that [HDFS-15547|https://issues.apache.org/jira/browse/HDFS-15547] [HDFS-15547] has solved a similar problem, which is a good feature. > HDFS tiered storage support > --- > > Key: HDFS-16009 > URL: https://issues.apache.org/jira/browse/HDFS-16009 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Priority: Major > > HDFS currently stores data blocks on different data nodes. These data nodes > are random within the scope of the network topology (local rack/different > racks/node group). > When the NameNode allocates DataNodes to the Block, it is randomly selected, > that is, each DataNode is selected equally. > We should make a hierarchical strategy. For example, DataNode1, DataNode2, > and DataNode3 are the same level, and DataNode4 and DataNode5 are the same > level. Because the hardware between DataNodes may be different, in this case, > it is necessary for us to do tiered storage, so that DataNodes with poor > hardware can store up to 50%, and 40% or even lower. This is conducive to > cluster maintenance and stability. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16010) Improve NamenodeBeanMetrics#getSafemode() to print IOException
[ https://issues.apache.org/jira/browse/HDFS-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangHua Zhu updated HDFS-16010: Labels: pull-request-available (was: ) > Improve NamenodeBeanMetrics#getSafemode() to print IOException > -- > > Key: HDFS-16010 > URL: https://issues.apache.org/jira/browse/HDFS-16010 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Attachments: image-2021-05-06-20-34-58-981.png > > > When executing NamenodeBeanMetrics#getSafemode(), if you encounter > IOException, missing spaces in the log information. > !image-2021-05-06-20-34-58-981.png! > In addition, we should also size the information present in the IOException, > which is useful. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16008) RBF: Tool to initialize ViewFS Mapping to Router
[
https://issues.apache.org/jira/browse/HDFS-16008?focusedWorklogId=593124=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-593124
]
ASF GitHub Bot logged work on HDFS-16008:
-
Author: ASF GitHub Bot
Created on: 07/May/21 03:14
Start Date: 07/May/21 03:14
Worklog Time Spent: 10m
Work Description: zhuxiangyi commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r627895751
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##
@@ -1035,6 +1054,65 @@ private boolean updateQuota(String mount, long nsQuota,
long ssQuota)
.updateMountTableEntry(updateRequest);
return updateResponse.getStatus();
}
+
+ /**
+ * initViewFsToMountTable.
+ * @param clusterName The specified cluster to initialize.
+ * @return If the quota was updated.
+ * @throws IOException Error adding the mount point.
+ */
+ public boolean initViewFsToMountTable(String clusterName)
+ throws IOException {
+// fs.viewfs.mounttable.ClusterX.link./data
+final String mountTablePrefix =
+Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+Constants.CONFIG_VIEWFS_LINK + "./";
+Map viewFsMap = getConf().getValByRegex(mountTablePrefix);
+if (viewFsMap.isEmpty()) {
+ System.out.println("There is no ViewFs mapping to initialize.");
+ return true;
+}
+for (Entry entry : viewFsMap.entrySet()) {
+ Path path = new Path(entry.getValue());
+ DestinationOrder order = DestinationOrder.HASH;
+ String[] mount = entry.getKey().split(
+ clusterName + "." + Constants.CONFIG_VIEWFS_LINK + ".");
+ if (mount.length < 2) {
+System.out.println("Added Mount Point failed " + entry.getKey());
+continue;
+ }
+ String[] nss = new String[]{path.toUri().getAuthority()};
+ boolean added = addMount(
+ mount[1], nss, path.toUri().getPath(), false,
+ false, order, getACLEntityFormHdfsPath(path));
+ if (added) {
+System.out.println("added mount point " + mount[1]);
+ }
+}
+return true;
+ }
+
+ /**
+ * Returns ACLEntity according to a HDFS pat.
+ * @param path A path of HDFS.
+ */
+ public ACLEntity getACLEntityFormHdfsPath(Path path){
+String owner = null;
+String group = null;
+FsPermission mode = null;
+try {
+ FileSystem fs = path.getFileSystem(getConf());
+ if (fs.exists(path)) {
+FileStatus fileStatus = fs.getFileStatus(path);
+owner = fileStatus.getOwner();
+group = fileStatus.getGroup();
+mode = fileStatus.getPermission();
+ }
+} catch (IOException e) {
+ System.out.println("Exception encountered " + e);
Review comment:
Yes,thank you for reminding me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 593124)
Time Spent: 40m (was: 0.5h)
> RBF: Tool to initialize ViewFS Mapping to Router
>
>
> Key: HDFS-16008
> URL: https://issues.apache.org/jira/browse/HDFS-16008
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.3.1
>Reporter: zhu
>Assignee: zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> This is a tool for initializing ViewFS Mapping to Router.
> Some companies are currently migrating from viewfs to router, I think they
> need this tool.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16008) RBF: Tool to initialize ViewFS Mapping to Router
[
https://issues.apache.org/jira/browse/HDFS-16008?focusedWorklogId=593123=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-593123
]
ASF GitHub Bot logged work on HDFS-16008:
-
Author: ASF GitHub Bot
Created on: 07/May/21 03:14
Start Date: 07/May/21 03:14
Worklog Time Spent: 10m
Work Description: zhuxiangyi commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r627895751
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##
@@ -1035,6 +1054,65 @@ private boolean updateQuota(String mount, long nsQuota,
long ssQuota)
.updateMountTableEntry(updateRequest);
return updateResponse.getStatus();
}
+
+ /**
+ * initViewFsToMountTable.
+ * @param clusterName The specified cluster to initialize.
+ * @return If the quota was updated.
+ * @throws IOException Error adding the mount point.
+ */
+ public boolean initViewFsToMountTable(String clusterName)
+ throws IOException {
+// fs.viewfs.mounttable.ClusterX.link./data
+final String mountTablePrefix =
+Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+Constants.CONFIG_VIEWFS_LINK + "./";
+Map viewFsMap = getConf().getValByRegex(mountTablePrefix);
+if (viewFsMap.isEmpty()) {
+ System.out.println("There is no ViewFs mapping to initialize.");
+ return true;
+}
+for (Entry entry : viewFsMap.entrySet()) {
+ Path path = new Path(entry.getValue());
+ DestinationOrder order = DestinationOrder.HASH;
+ String[] mount = entry.getKey().split(
+ clusterName + "." + Constants.CONFIG_VIEWFS_LINK + ".");
+ if (mount.length < 2) {
+System.out.println("Added Mount Point failed " + entry.getKey());
+continue;
+ }
+ String[] nss = new String[]{path.toUri().getAuthority()};
+ boolean added = addMount(
+ mount[1], nss, path.toUri().getPath(), false,
+ false, order, getACLEntityFormHdfsPath(path));
+ if (added) {
+System.out.println("added mount point " + mount[1]);
+ }
+}
+return true;
+ }
+
+ /**
+ * Returns ACLEntity according to a HDFS pat.
+ * @param path A path of HDFS.
+ */
+ public ACLEntity getACLEntityFormHdfsPath(Path path){
+String owner = null;
+String group = null;
+FsPermission mode = null;
+try {
+ FileSystem fs = path.getFileSystem(getConf());
+ if (fs.exists(path)) {
+FileStatus fileStatus = fs.getFileStatus(path);
+owner = fileStatus.getOwner();
+group = fileStatus.getGroup();
+mode = fileStatus.getPermission();
+ }
+} catch (IOException e) {
+ System.out.println("Exception encountered " + e);
Review comment:
Yes,Thank you for reminding me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 593123)
Time Spent: 0.5h (was: 20m)
> RBF: Tool to initialize ViewFS Mapping to Router
>
>
> Key: HDFS-16008
> URL: https://issues.apache.org/jira/browse/HDFS-16008
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.3.1
>Reporter: zhu
>Assignee: zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> This is a tool for initializing ViewFS Mapping to Router.
> Some companies are currently migrating from viewfs to router, I think they
> need this tool.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16008) RBF: Tool to initialize ViewFS Mapping to Router
[
https://issues.apache.org/jira/browse/HDFS-16008?focusedWorklogId=593118=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-593118
]
ASF GitHub Bot logged work on HDFS-16008:
-
Author: ASF GitHub Bot
Created on: 07/May/21 03:01
Start Date: 07/May/21 03:01
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r627891890
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##
@@ -1035,6 +1054,65 @@ private boolean updateQuota(String mount, long nsQuota,
long ssQuota)
.updateMountTableEntry(updateRequest);
return updateResponse.getStatus();
}
+
+ /**
+ * initViewFsToMountTable.
+ * @param clusterName The specified cluster to initialize.
+ * @return If the quota was updated.
+ * @throws IOException Error adding the mount point.
+ */
+ public boolean initViewFsToMountTable(String clusterName)
+ throws IOException {
+// fs.viewfs.mounttable.ClusterX.link./data
+final String mountTablePrefix =
+Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+Constants.CONFIG_VIEWFS_LINK + "./";
+Map viewFsMap = getConf().getValByRegex(mountTablePrefix);
+if (viewFsMap.isEmpty()) {
+ System.out.println("There is no ViewFs mapping to initialize.");
+ return true;
+}
+for (Entry entry : viewFsMap.entrySet()) {
+ Path path = new Path(entry.getValue());
+ DestinationOrder order = DestinationOrder.HASH;
+ String[] mount = entry.getKey().split(
+ clusterName + "." + Constants.CONFIG_VIEWFS_LINK + ".");
+ if (mount.length < 2) {
+System.out.println("Added Mount Point failed " + entry.getKey());
+continue;
+ }
+ String[] nss = new String[]{path.toUri().getAuthority()};
+ boolean added = addMount(
+ mount[1], nss, path.toUri().getPath(), false,
+ false, order, getACLEntityFormHdfsPath(path));
+ if (added) {
+System.out.println("added mount point " + mount[1]);
+ }
+}
+return true;
+ }
+
+ /**
+ * Returns ACLEntity according to a HDFS pat.
+ * @param path A path of HDFS.
+ */
+ public ACLEntity getACLEntityFormHdfsPath(Path path){
+String owner = null;
+String group = null;
+FsPermission mode = null;
+try {
+ FileSystem fs = path.getFileSystem(getConf());
+ if (fs.exists(path)) {
+FileStatus fileStatus = fs.getFileStatus(path);
+owner = fileStatus.getOwner();
+group = fileStatus.getGroup();
+mode = fileStatus.getPermission();
+ }
+} catch (IOException e) {
+ System.out.println("Exception encountered " + e);
Review comment:
Yes, but it should go to System.err not System.out, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 593118)
Time Spent: 20m (was: 10m)
> RBF: Tool to initialize ViewFS Mapping to Router
>
>
> Key: HDFS-16008
> URL: https://issues.apache.org/jira/browse/HDFS-16008
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.3.1
>Reporter: zhu
>Assignee: zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This is a tool for initializing ViewFS Mapping to Router.
> Some companies are currently migrating from viewfs to router, I think they
> need this tool.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16008) RBF: Tool to initialize ViewFS Mapping to Router
[ https://issues.apache.org/jira/browse/HDFS-16008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16008: -- Labels: pull-request-available (was: ) > RBF: Tool to initialize ViewFS Mapping to Router > > > Key: HDFS-16008 > URL: https://issues.apache.org/jira/browse/HDFS-16008 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.3.1 >Reporter: zhu >Assignee: zhu >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This is a tool for initializing ViewFS Mapping to Router. > Some companies are currently migrating from viewfs to router, I think they > need this tool. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16008) RBF: Tool to initialize ViewFS Mapping to Router
[
https://issues.apache.org/jira/browse/HDFS-16008?focusedWorklogId=593116=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-593116
]
ASF GitHub Bot logged work on HDFS-16008:
-
Author: ASF GitHub Bot
Created on: 07/May/21 02:34
Start Date: 07/May/21 02:34
Worklog Time Spent: 10m
Work Description: zhuxiangyi commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r627882834
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##
@@ -1035,6 +1054,65 @@ private boolean updateQuota(String mount, long nsQuota,
long ssQuota)
.updateMountTableEntry(updateRequest);
return updateResponse.getStatus();
}
+
+ /**
+ * initViewFsToMountTable.
+ * @param clusterName The specified cluster to initialize.
+ * @return If the quota was updated.
+ * @throws IOException Error adding the mount point.
+ */
+ public boolean initViewFsToMountTable(String clusterName)
+ throws IOException {
+// fs.viewfs.mounttable.ClusterX.link./data
+final String mountTablePrefix =
+Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+Constants.CONFIG_VIEWFS_LINK + "./";
+Map viewFsMap = getConf().getValByRegex(mountTablePrefix);
+if (viewFsMap.isEmpty()) {
+ System.out.println("There is no ViewFs mapping to initialize.");
+ return true;
+}
+for (Entry entry : viewFsMap.entrySet()) {
+ Path path = new Path(entry.getValue());
+ DestinationOrder order = DestinationOrder.HASH;
+ String[] mount = entry.getKey().split(
+ clusterName + "." + Constants.CONFIG_VIEWFS_LINK + ".");
+ if (mount.length < 2) {
+System.out.println("Added Mount Point failed " + entry.getKey());
+continue;
+ }
+ String[] nss = new String[]{path.toUri().getAuthority()};
+ boolean added = addMount(
+ mount[1], nss, path.toUri().getPath(), false,
+ false, order, getACLEntityFormHdfsPath(path));
+ if (added) {
+System.out.println("added mount point " + mount[1]);
+ }
+}
+return true;
+ }
+
+ /**
+ * Returns ACLEntity according to a HDFS pat.
+ * @param path A path of HDFS.
+ */
+ public ACLEntity getACLEntityFormHdfsPath(Path path){
+String owner = null;
+String group = null;
+FsPermission mode = null;
+try {
+ FileSystem fs = path.getFileSystem(getConf());
+ if (fs.exists(path)) {
+FileStatus fileStatus = fs.getFileStatus(path);
+owner = fileStatus.getOwner();
+group = fileStatus.getGroup();
+mode = fileStatus.getPermission();
+ }
+} catch (IOException e) {
+ System.out.println("Exception encountered " + e);
Review comment:
I think it fails to get FileStatus and can continue to use the default
ACLEntity to add mapping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 593116)
Remaining Estimate: 0h
Time Spent: 10m
> RBF: Tool to initialize ViewFS Mapping to Router
>
>
> Key: HDFS-16008
> URL: https://issues.apache.org/jira/browse/HDFS-16008
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.3.1
>Reporter: zhu
>Assignee: zhu
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is a tool for initializing ViewFS Mapping to Router.
> Some companies are currently migrating from viewfs to router, I think they
> need this tool.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16007) Vulnerabilities found when serializing enum value
[
https://issues.apache.org/jira/browse/HDFS-16007?focusedWorklogId=593086=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-593086
]
ASF GitHub Bot logged work on HDFS-16007:
-
Author: ASF GitHub Bot
Created on: 07/May/21 00:20
Start Date: 07/May/21 00:20
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2982:
URL: https://github.com/apache/hadoop/pull/2982#issuecomment-833965963
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 29s | | trunk passed |
| +1 :green_heart: | compile | 1m 36s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 34s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 54s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 46s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 337m 27s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2982/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 435m 49s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.datanode.TestIncrementalBrVariations |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.TestStateAlignmentContextWithHA |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
[jira] [Updated] (HDFS-16007) Vulnerabilities found when serializing enum value
[ https://issues.apache.org/jira/browse/HDFS-16007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16007: -- Labels: pull-request-available (was: ) > Vulnerabilities found when serializing enum value > - > > Key: HDFS-16007 > URL: https://issues.apache.org/jira/browse/HDFS-16007 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: junwen yang >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > ReplicaState enum is using ordinal to conduct serialization and > deserialization, which is vulnerable to the order, to cause issues similar to > HDFS-15624. > To avoid it, either adding comments to let later developer not to change this > enum, or add index checking in the read and getState function to avoid index > out of bound error. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16007) Vulnerabilities found when serializing enum value
[
https://issues.apache.org/jira/browse/HDFS-16007?focusedWorklogId=593081=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-593081
]
ASF GitHub Bot logged work on HDFS-16007:
-
Author: ASF GitHub Bot
Created on: 07/May/21 00:09
Start Date: 07/May/21 00:09
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2982:
URL: https://github.com/apache/hadoop/pull/2982#issuecomment-833960045
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 30s | | trunk passed |
| +1 :green_heart: | compile | 1m 38s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 1s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 34s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 47s | | trunk passed |
| -1 :x: | shadedclient | 6m 56s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 29s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2982/3/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 0m 23s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2982/3/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | javac | 0m 23s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2982/3/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04. |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2982/3/artifact/out/blanks-eol.txt)
| The patch has 40 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | blanks | 0m 0s |
[/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2982/3/artifact/out/blanks-tabs.txt)
| The patch 600 line(s) with tabs. |
| +1 :green_heart: | checkstyle | 1m 2s | | the patch passed |
| -1 :x: | mvnsite | 1m 22s |
[/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2982/3/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | javadoc | 0m 11s |
[/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2982/3/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04. |
| +1 :green_heart: | javadoc | 1m 25s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 241m 27s |
[jira] [Commented] (HDFS-16009) HDFS tiered storage support
[ https://issues.apache.org/jira/browse/HDFS-16009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340454#comment-17340454 ] Leon Gao commented on HDFS-16009: - Hi [~jianghuazhu], maybe HDFS-15683 can solve the problem? You can configure capacity ratio for volumes and storage types. > HDFS tiered storage support > --- > > Key: HDFS-16009 > URL: https://issues.apache.org/jira/browse/HDFS-16009 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Priority: Major > > HDFS currently stores data blocks on different data nodes. These data nodes > are random within the scope of the network topology (local rack/different > racks/node group). > When the NameNode allocates DataNodes to the Block, it is randomly selected, > that is, each DataNode is selected equally. > We should make a hierarchical strategy. For example, DataNode1, DataNode2, > and DataNode3 are the same level, and DataNode4 and DataNode5 are the same > level. Because the hardware between DataNodes may be different, in this case, > it is necessary for us to do tiered storage, so that DataNodes with poor > hardware can store up to 50%, and 40% or even lower. This is conducive to > cluster maintenance and stability. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15982) Deleted data using HTTP API should be saved to the trash
[
https://issues.apache.org/jira/browse/HDFS-15982?focusedWorklogId=592978=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-592978
]
ASF GitHub Bot logged work on HDFS-15982:
-
Author: ASF GitHub Bot
Created on: 06/May/21 19:06
Start Date: 06/May/21 19:06
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2976:
URL: https://github.com/apache/hadoop/pull/2976#issuecomment-833787216
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | jshint | 0m 1s | | jshint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 6s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 13s | | trunk passed |
| +1 :green_heart: | compile | 4m 52s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 4m 30s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 43s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 28s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 7m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 57s | | the patch passed |
| +1 :green_heart: | compile | 5m 6s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 5m 6s | | the patch passed |
| +1 :green_heart: | compile | 4m 56s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 4m 56s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 9s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 12s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 13s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 57s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 8m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 21s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 235m 3s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2976/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 6m 11s | | hadoop-hdfs-httpfs in the patch
passed. |
| +1 :green_heart: | unit | 18m 14s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 389m 31s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2976/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2976 |
| Optional Tests | dupname asflicense compile javac
[jira] [Commented] (HDFS-16007) Vulnerabilities found when serializing enum value
[ https://issues.apache.org/jira/browse/HDFS-16007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340370#comment-17340370 ] junwen yang commented on HDFS-16007: [~vjasani] Looks perfect! > Vulnerabilities found when serializing enum value > - > > Key: HDFS-16007 > URL: https://issues.apache.org/jira/browse/HDFS-16007 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: junwen yang >Assignee: Viraj Jasani >Priority: Major > > ReplicaState enum is using ordinal to conduct serialization and > deserialization, which is vulnerable to the order, to cause issues similar to > HDFS-15624. > To avoid it, either adding comments to let later developer not to change this > enum, or add index checking in the read and getState function to avoid index > out of bound error. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16007) Vulnerabilities found when serializing enum value
[
https://issues.apache.org/jira/browse/HDFS-16007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340368#comment-17340368
]
Viraj Jasani commented on HDFS-16007:
-
[~jwjwyoung] Would you like to take a look at PR? In addition to small
improvement, I have added this comment:
{code:java}
// Since ReplicaState (de)serialization depends on ordinal, either adding
// new value should be avoided to this enum or newly appended value should
// be handled by NameNodeLayoutVersion#Feature.
{code}
> Vulnerabilities found when serializing enum value
> -
>
> Key: HDFS-16007
> URL: https://issues.apache.org/jira/browse/HDFS-16007
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: junwen yang
>Assignee: Viraj Jasani
>Priority: Major
>
> ReplicaState enum is using ordinal to conduct serialization and
> deserialization, which is vulnerable to the order, to cause issues similar to
> HDFS-15624.
> To avoid it, either adding comments to let later developer not to change this
> enum, or add index checking in the read and getState function to avoid index
> out of bound error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15923) RBF: Authentication failed when rename accross sub clusters
[
https://issues.apache.org/jira/browse/HDFS-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340343#comment-17340343
]
Íñigo Goiri commented on HDFS-15923:
+1 on [^HDFS-15923.003.patch].
> RBF: Authentication failed when rename accross sub clusters
>
>
> Key: HDFS-15923
> URL: https://issues.apache.org/jira/browse/HDFS-15923
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: zhuobin zheng
>Assignee: zhuobin zheng
>Priority: Major
> Labels: RBF, pull-request-available, rename
> Attachments: HDFS-15923.001.patch, HDFS-15923.002.patch,
> HDFS-15923.003.patch, HDFS-15923.stack-trace,
> hdfs-15923-fix-security-issue.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Rename accross subcluster with RBF and Kerberos environment. Will encounter
> the following two errors:
> # Save Object to journal.
> # Precheck try to get src file status
> So, we need use Router Login UGI doAs create DistcpProcedure and
> TrashProcedure and submit Job.
>
> Beside, we should check user permission for src and dst path in router side
> before do rename internal. (HDFS-15973)
> First: Save Object to journal.
> {code:java}
> // code placeholder
> 2021-03-23 14:01:16,233 WARN org.apache.hadoop.ipc.Client: Exception
> encountered while connecting to the server
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:408)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:622)
> at
> org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:413)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:822)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:818)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:818)
> at
> org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1636)
> at org.apache.hadoop.ipc.Client.call(Client.java:1452)
> at org.apache.hadoop.ipc.Client.call(Client.java:1405)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy11.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:376)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy12.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:277)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1240)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1219)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1201)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1139)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:533)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:530)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
>
[jira] [Commented] (HDFS-16007) Vulnerabilities found when serializing enum value
[ https://issues.apache.org/jira/browse/HDFS-16007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340248#comment-17340248 ] junwen yang commented on HDFS-16007: [~vjasani] yes, adding the index information is what I mean. Since if in the new version, a new value is appended, then the index will be the original length + 1, if it's sent to the old version, it will cause the arrayoutofbound error. > Vulnerabilities found when serializing enum value > - > > Key: HDFS-16007 > URL: https://issues.apache.org/jira/browse/HDFS-16007 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: junwen yang >Assignee: Viraj Jasani >Priority: Major > > ReplicaState enum is using ordinal to conduct serialization and > deserialization, which is vulnerable to the order, to cause issues similar to > HDFS-15624. > To avoid it, either adding comments to let later developer not to change this > enum, or add index checking in the read and getState function to avoid index > out of bound error. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-16010) Improve NamenodeBeanMetrics#getSafemode() to print IOException
[ https://issues.apache.org/jira/browse/HDFS-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-16010 started by JiangHua Zhu. --- > Improve NamenodeBeanMetrics#getSafemode() to print IOException > -- > > Key: HDFS-16010 > URL: https://issues.apache.org/jira/browse/HDFS-16010 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Attachments: image-2021-05-06-20-34-58-981.png > > > When executing NamenodeBeanMetrics#getSafemode(), if you encounter > IOException, missing spaces in the log information. > !image-2021-05-06-20-34-58-981.png! > In addition, we should also size the information present in the IOException, > which is useful. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16010) Improve NamenodeBeanMetrics#getSafemode() to print IOException
[ https://issues.apache.org/jira/browse/HDFS-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangHua Zhu updated HDFS-16010: Description: When executing NamenodeBeanMetrics#getSafemode(), if you encounter IOException, missing spaces in the log information. !image-2021-05-06-20-34-58-981.png! In addition, we should also size the information present in the IOException, which is useful. was: When executing NamenodeBeanMetrics#getSafemode(), if you encounter IOException, missing spaces in the log information. !image-2021-05-06-20-34-58-981.png! > Improve NamenodeBeanMetrics#getSafemode() to print IOException > -- > > Key: HDFS-16010 > URL: https://issues.apache.org/jira/browse/HDFS-16010 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Attachments: image-2021-05-06-20-34-58-981.png > > > When executing NamenodeBeanMetrics#getSafemode(), if you encounter > IOException, missing spaces in the log information. > !image-2021-05-06-20-34-58-981.png! > In addition, we should also size the information present in the IOException, > which is useful. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16010) Improve NamenodeBeanMetrics#getSafemode() to print IOException
[ https://issues.apache.org/jira/browse/HDFS-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangHua Zhu reassigned HDFS-16010: --- Assignee: JiangHua Zhu > Improve NamenodeBeanMetrics#getSafemode() to print IOException > -- > > Key: HDFS-16010 > URL: https://issues.apache.org/jira/browse/HDFS-16010 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Attachments: image-2021-05-06-20-34-58-981.png > > > When executing NamenodeBeanMetrics#getSafemode(), if you encounter > IOException, missing spaces in the log information. > !image-2021-05-06-20-34-58-981.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16010) Improve NamenodeBeanMetrics#getSafemode() to print IOException
JiangHua Zhu created HDFS-16010: --- Summary: Improve NamenodeBeanMetrics#getSafemode() to print IOException Key: HDFS-16010 URL: https://issues.apache.org/jira/browse/HDFS-16010 Project: Hadoop HDFS Issue Type: Improvement Reporter: JiangHua Zhu Attachments: image-2021-05-06-20-34-58-981.png When executing NamenodeBeanMetrics#getSafemode(), if you encounter IOException, missing spaces in the log information. !image-2021-05-06-20-34-58-981.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15969) DFSClient prints token information a string format
[
https://issues.apache.org/jira/browse/HDFS-15969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340142#comment-17340142
]
Takanobu Asanuma commented on HDFS-15969:
-
The following is an example of the token information.
{noformat}
Created token for tasanuma: HDFS_DELEGATION_TOKEN owner=tasanuma,
renewer=JobTracker, realUser=, issueDate=1620293228047, maxDate=1620898028047,
sequenceNumber=1, masterKeyId=2 on ha-hdfs:ha-nn-uri-0
{noformat}
I have seen this information in several other logs as well. I don't think it
has ever been considered sensitive in the Hadoop community. Given the impact on
existing users, we may want to keep it as INFO level.
> DFSClient prints token information a string format
> ---
>
> Key: HDFS-15969
> URL: https://issues.apache.org/jira/browse/HDFS-15969
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Bhavik Patel
>Assignee: Bhavik Patel
>Priority: Minor
> Attachments: HDFS-15969.001.patch
>
>
> DFSclient prints token information in a string format, as this is sensitive
> information it must be moved to debug level or can be exempted even from
> debug level
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-16009) HDFS tiered storage support
[ https://issues.apache.org/jira/browse/HDFS-16009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340125#comment-17340125 ] JiangHua Zhu edited comment on HDFS-16009 at 5/6/21, 9:57 AM: -- [~weichiu], [~kihwal], do you have some good suggestions. was (Author: jianghuazhu): Wei-Chiu Chuang, Kihwal Lee, do you have some good suggestions. > HDFS tiered storage support > --- > > Key: HDFS-16009 > URL: https://issues.apache.org/jira/browse/HDFS-16009 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Priority: Major > > HDFS currently stores data blocks on different data nodes. These data nodes > are random within the scope of the network topology (local rack/different > racks/node group). > When the NameNode allocates DataNodes to the Block, it is randomly selected, > that is, each DataNode is selected equally. > We should make a hierarchical strategy. For example, DataNode1, DataNode2, > and DataNode3 are the same level, and DataNode4 and DataNode5 are the same > level. Because the hardware between DataNodes may be different, in this case, > it is necessary for us to do tiered storage, so that DataNodes with poor > hardware can store up to 50%, and 40% or even lower. This is conducive to > cluster maintenance and stability. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16009) HDFS tiered storage support
[ https://issues.apache.org/jira/browse/HDFS-16009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340125#comment-17340125 ] JiangHua Zhu commented on HDFS-16009: - Wei-Chiu Chuang, Kihwal Lee, do you have some good suggestions. > HDFS tiered storage support > --- > > Key: HDFS-16009 > URL: https://issues.apache.org/jira/browse/HDFS-16009 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Priority: Major > > HDFS currently stores data blocks on different data nodes. These data nodes > are random within the scope of the network topology (local rack/different > racks/node group). > When the NameNode allocates DataNodes to the Block, it is randomly selected, > that is, each DataNode is selected equally. > We should make a hierarchical strategy. For example, DataNode1, DataNode2, > and DataNode3 are the same level, and DataNode4 and DataNode5 are the same > level. Because the hardware between DataNodes may be different, in this case, > it is necessary for us to do tiered storage, so that DataNodes with poor > hardware can store up to 50%, and 40% or even lower. This is conducive to > cluster maintenance and stability. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16009) HDFS tiered storage support
[ https://issues.apache.org/jira/browse/HDFS-16009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340122#comment-17340122 ] JiangHua Zhu commented on HDFS-16009: - What I said here is not the same as [link title|https://issues.apache.org/jira/browse/HDFS-9411]. > HDFS tiered storage support > --- > > Key: HDFS-16009 > URL: https://issues.apache.org/jira/browse/HDFS-16009 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Priority: Major > > HDFS currently stores data blocks on different data nodes. These data nodes > are random within the scope of the network topology (local rack/different > racks/node group). > When the NameNode allocates DataNodes to the Block, it is randomly selected, > that is, each DataNode is selected equally. > We should make a hierarchical strategy. For example, DataNode1, DataNode2, > and DataNode3 are the same level, and DataNode4 and DataNode5 are the same > level. Because the hardware between DataNodes may be different, in this case, > it is necessary for us to do tiered storage, so that DataNodes with poor > hardware can store up to 50%, and 40% or even lower. This is conducive to > cluster maintenance and stability. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16007) Vulnerabilities found when serializing enum value
[ https://issues.apache.org/jira/browse/HDFS-16007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HDFS-16007: Status: Patch Available (was: In Progress) > Vulnerabilities found when serializing enum value > - > > Key: HDFS-16007 > URL: https://issues.apache.org/jira/browse/HDFS-16007 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: junwen yang >Assignee: Viraj Jasani >Priority: Major > > ReplicaState enum is using ordinal to conduct serialization and > deserialization, which is vulnerable to the order, to cause issues similar to > HDFS-15624. > To avoid it, either adding comments to let later developer not to change this > enum, or add index checking in the read and getState function to avoid index > out of bound error. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-16009) HDFS tiered storage support
[ https://issues.apache.org/jira/browse/HDFS-16009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340122#comment-17340122 ] JiangHua Zhu edited comment on HDFS-16009 at 5/6/21, 9:51 AM: -- What I said here is not the same as [HDFS-9411|https://issues.apache.org/jira/browse/HDFS-9411]. was (Author: jianghuazhu): What I said here is not the same as [link title|https://issues.apache.org/jira/browse/HDFS-9411]. > HDFS tiered storage support > --- > > Key: HDFS-16009 > URL: https://issues.apache.org/jira/browse/HDFS-16009 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Priority: Major > > HDFS currently stores data blocks on different data nodes. These data nodes > are random within the scope of the network topology (local rack/different > racks/node group). > When the NameNode allocates DataNodes to the Block, it is randomly selected, > that is, each DataNode is selected equally. > We should make a hierarchical strategy. For example, DataNode1, DataNode2, > and DataNode3 are the same level, and DataNode4 and DataNode5 are the same > level. Because the hardware between DataNodes may be different, in this case, > it is necessary for us to do tiered storage, so that DataNodes with poor > hardware can store up to 50%, and 40% or even lower. This is conducive to > cluster maintenance and stability. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16007) Vulnerabilities found when serializing enum value
[ https://issues.apache.org/jira/browse/HDFS-16007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HDFS-16007: Target Version/s: 3.3.1, 3.4.0, 3.1.5, 3.2.3 > Vulnerabilities found when serializing enum value > - > > Key: HDFS-16007 > URL: https://issues.apache.org/jira/browse/HDFS-16007 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: junwen yang >Assignee: Viraj Jasani >Priority: Major > > ReplicaState enum is using ordinal to conduct serialization and > deserialization, which is vulnerable to the order, to cause issues similar to > HDFS-15624. > To avoid it, either adding comments to let later developer not to change this > enum, or add index checking in the read and getState function to avoid index > out of bound error. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-16007) Vulnerabilities found when serializing enum value
[ https://issues.apache.org/jira/browse/HDFS-16007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-16007 started by Viraj Jasani. --- > Vulnerabilities found when serializing enum value > - > > Key: HDFS-16007 > URL: https://issues.apache.org/jira/browse/HDFS-16007 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: junwen yang >Assignee: Viraj Jasani >Priority: Major > > ReplicaState enum is using ordinal to conduct serialization and > deserialization, which is vulnerable to the order, to cause issues similar to > HDFS-15624. > To avoid it, either adding comments to let later developer not to change this > enum, or add index checking in the read and getState function to avoid index > out of bound error. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16009) HDFS tiered storage support
JiangHua Zhu created HDFS-16009: --- Summary: HDFS tiered storage support Key: HDFS-16009 URL: https://issues.apache.org/jira/browse/HDFS-16009 Project: Hadoop HDFS Issue Type: Improvement Reporter: JiangHua Zhu HDFS currently stores data blocks on different data nodes. These data nodes are random within the scope of the network topology (local rack/different racks/node group). When the NameNode allocates DataNodes to the Block, it is randomly selected, that is, each DataNode is selected equally. We should make a hierarchical strategy. For example, DataNode1, DataNode2, and DataNode3 are the same level, and DataNode4 and DataNode5 are the same level. Because the hardware between DataNodes may be different, in this case, it is necessary for us to do tiered storage, so that DataNodes with poor hardware can store up to 50%, and 40% or even lower. This is conducive to cluster maintenance and stability. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16007) Vulnerabilities found when serializing enum value
[ https://issues.apache.org/jira/browse/HDFS-16007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani reassigned HDFS-16007: --- Assignee: Viraj Jasani > Vulnerabilities found when serializing enum value > - > > Key: HDFS-16007 > URL: https://issues.apache.org/jira/browse/HDFS-16007 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: junwen yang >Assignee: Viraj Jasani >Priority: Major > > ReplicaState enum is using ordinal to conduct serialization and > deserialization, which is vulnerable to the order, to cause issues similar to > HDFS-15624. > To avoid it, either adding comments to let later developer not to change this > enum, or add index checking in the read and getState function to avoid index > out of bound error. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16007) Vulnerabilities found when serializing enum value
[
https://issues.apache.org/jira/browse/HDFS-16007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340080#comment-17340080
]
Viraj Jasani commented on HDFS-16007:
-
For any new ReplicaState introduced in future, if we just append value after
last one (TEMPORARY(4) as of today), it should work fine right?
{quote}add index checking in the read and getState function to avoid index out
of bound error.
{quote}
Are you suggesting this so that we can throw simple error message, something
like "Provided index should be in the range of: 0 to cachedValues.length-1"? If
so, then it makes sense.
> Vulnerabilities found when serializing enum value
> -
>
> Key: HDFS-16007
> URL: https://issues.apache.org/jira/browse/HDFS-16007
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: junwen yang
>Priority: Major
>
> ReplicaState enum is using ordinal to conduct serialization and
> deserialization, which is vulnerable to the order, to cause issues similar to
> HDFS-15624.
> To avoid it, either adding comments to let later developer not to change this
> enum, or add index checking in the read and getState function to avoid index
> out of bound error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15788) Correct the statement for pmem cache to reflect cache persistence support
[ https://issues.apache.org/jira/browse/HDFS-15788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340051#comment-17340051 ] Feilong He commented on HDFS-15788: --- This is not a critical issue. I just changed the target version to just 3.4.0. > Correct the statement for pmem cache to reflect cache persistence support > - > > Key: HDFS-15788 > URL: https://issues.apache.org/jira/browse/HDFS-15788 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Minor > Attachments: HDFS-15788-01.patch, HDFS-15788-02.patch > > > Correct the statement for pmem cache to reflect cache persistence support. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15788) Correct the statement for pmem cache to reflect cache persistence support
[ https://issues.apache.org/jira/browse/HDFS-15788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Feilong He updated HDFS-15788: -- Target Version/s: 3.4.0 (was: 3.3.1, 3.4.0) > Correct the statement for pmem cache to reflect cache persistence support > - > > Key: HDFS-15788 > URL: https://issues.apache.org/jira/browse/HDFS-15788 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Minor > Attachments: HDFS-15788-01.patch, HDFS-15788-02.patch > > > Correct the statement for pmem cache to reflect cache persistence support. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
