[
https://issues.apache.org/jira/browse/HDFS-17602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongkun Wu updated HDFS-17602:
-------------------------------
Description:
We have seventy thousand nodes in our production cluster, we use rbf to do
namespace federation. when uses space order in rbf, and writes to the mount
point, it will fail to write and generated an empty file.
!image-2024-08-12-10-08-58-271.png!
In the
MultipleDestinationMountTableResolver.java we invoke
orderedResolver.getFirstNamespace(path, mountTableResult);
It will then invoke this function in RouterResolver.java
!image-2024-08-12-10-12-48-428.png!
and now we are in
chooseFirstNamespace function in AvailableSpaceResolver.java
!image-2024-08-12-10-14-20-580.png!
The path parameter is the destination where we want to create a file
the loc parameter is the mount point we set
this function will choose the most available namespace in all the namespace we
have in StateStore, which is not the same as the mount point we set for our
destination.
As a result we will get a namespace irrelevant to the namespaces we set for
the destination path
!image-2024-08-12-10-25-42-863.png!
in the log above:
we get the namespace we don't set with our destination path, So the it will
choose the first namespace it sees and it's not really the most available
namespace among the namespaces we set for our destination
was:
!image-2024-08-12-10-08-58-271.png!
In the
MultipleDestinationMountTableResolver.java we invoke
orderedResolver.getFirstNamespace(path, mountTableResult);
It will then invoke this function in RouterResolver.java
!image-2024-08-12-10-12-48-428.png!
and now we are in
chooseFirstNamespace function in AvailableSpaceResolver.java
!image-2024-08-12-10-14-20-580.png!
The path parameter is the destination where we want to create a file
the loc parameter is the mount point we set
this function will choose the most available namespace in all the namespace we
have in StateStore, which is not the same as the mount point we set for our
destination.
As a result we will get a namespace irrelevant to the namespaces we set for
the destination path
!image-2024-08-12-10-25-42-863.png!
in the log above:
we get the namespace we don't set with our destination path, So the it will
choose the first namespace it sees and it's not really the most available
namespace among the namespaces we set for our destination
> RBF mount point with SPACE order can not find the most available namespace,
> it will choose an irrelevant namespace
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17602
> URL: https://issues.apache.org/jira/browse/HDFS-17602
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: router
> Reporter: Zhongkun Wu
> Priority: Critical
> Labels: pull-request-available
> Attachments: image-2024-08-12-10-08-54-031.png,
> image-2024-08-12-10-08-58-271.png, image-2024-08-12-10-12-48-428.png,
> image-2024-08-12-10-14-20-580.png, image-2024-08-12-10-25-26-003.png,
> image-2024-08-12-10-25-42-863.png
>
>
> We have seventy thousand nodes in our production cluster, we use rbf to do
> namespace federation. when uses space order in rbf, and writes to the mount
> point, it will fail to write and generated an empty file.
>
>
>
> !image-2024-08-12-10-08-58-271.png!
> In the
> MultipleDestinationMountTableResolver.java we invoke
> orderedResolver.getFirstNamespace(path, mountTableResult);
> It will then invoke this function in RouterResolver.java
> !image-2024-08-12-10-12-48-428.png!
> and now we are in
> chooseFirstNamespace function in AvailableSpaceResolver.java
> !image-2024-08-12-10-14-20-580.png!
>
> The path parameter is the destination where we want to create a file
> the loc parameter is the mount point we set
>
> this function will choose the most available namespace in all the namespace
> we have in StateStore, which is not the same as the mount point we set for
> our destination.
>
> As a result we will get a namespace irrelevant to the namespaces we set for
> the destination path
>
> !image-2024-08-12-10-25-42-863.png!
> in the log above:
> we get the namespace we don't set with our destination path, So the it will
> choose the first namespace it sees and it's not really the most available
> namespace among the namespaces we set for our destination
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
