[jira] [Updated] (HDFS-10544) Balancer doesn't work with IPFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-10544:
--
Fix Version/s: (was: 2.9.0)
> Balancer doesn't work with IPFailoverProxyProvider
> --
>
> Key: HDFS-10544
> URL: https://issues.apache.org/jira/browse/HDFS-10544
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, ha
>Affects Versions: 2.6.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.6.5, 2.7.4, 3.0.0-alpha1
>
> Attachments: HDFS-10544-branch-2.7.patch, HDFS-10544.00.patch,
> HDFS-10544.01.patch, HDFS-10544.02.patch, HDFS-10544.03.patch,
> HDFS-10544.04.patch, HDFS-10544.05.patch
>
>
> Right now {{Balancer}} gets the NN URIs through
> {{DFSUtil#getNameServiceUris}}, which returns logical URIs in HA is enabled.
> If {{IPFailoverProxyProvider}} is used, {{Balancer}} will not be able to
> start.
> I think the bug is at {{DFSUtil#getNameServiceUris}}:
> {code}
> for (String nsId : getNameServiceIds(conf)) {
> if (HAUtil.isHAEnabled(conf, nsId)) {
> // Add the logical URI of the nameservice.
> try {
> ret.add(new URI(HdfsConstants.HDFS_URI_SCHEME + "://" + nsId));
> {code}
> Then {{if}} clause should also consider if the {{FailoverProxyProvider}} has
> {{useLogicalURI}} enabled. If not, {{getNameServiceUris}} should try to
> resolve the physical URI for this nsId.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10544) Balancer doesn't work with IPFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HDFS-10544:
---
Fix Version/s: (was: 2.7.3)
2.7.4
2.7.3 was already under process, changing the fix-version to 2.7.4.
> Balancer doesn't work with IPFailoverProxyProvider
> --
>
> Key: HDFS-10544
> URL: https://issues.apache.org/jira/browse/HDFS-10544
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, ha
>Affects Versions: 2.6.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.9.0, 2.6.5, 2.7.4, 3.0.0-alpha1
>
> Attachments: HDFS-10544-branch-2.7.patch, HDFS-10544.00.patch,
> HDFS-10544.01.patch, HDFS-10544.02.patch, HDFS-10544.03.patch,
> HDFS-10544.04.patch, HDFS-10544.05.patch
>
>
> Right now {{Balancer}} gets the NN URIs through
> {{DFSUtil#getNameServiceUris}}, which returns logical URIs in HA is enabled.
> If {{IPFailoverProxyProvider}} is used, {{Balancer}} will not be able to
> start.
> I think the bug is at {{DFSUtil#getNameServiceUris}}:
> {code}
> for (String nsId : getNameServiceIds(conf)) {
> if (HAUtil.isHAEnabled(conf, nsId)) {
> // Add the logical URI of the nameservice.
> try {
> ret.add(new URI(HdfsConstants.HDFS_URI_SCHEME + "://" + nsId));
> {code}
> Then {{if}} clause should also consider if the {{FailoverProxyProvider}} has
> {{useLogicalURI}} enabled. If not, {{getNameServiceUris}} should try to
> resolve the physical URI for this nsId.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10544) Balancer doesn't work with IPFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10544:
-
Component/s: ha
balancer & mover
> Balancer doesn't work with IPFailoverProxyProvider
> --
>
> Key: HDFS-10544
> URL: https://issues.apache.org/jira/browse/HDFS-10544
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, ha
>Affects Versions: 2.6.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.7.3, 2.9.0, 2.6.5, 3.0.0-alpha1
>
> Attachments: HDFS-10544-branch-2.7.patch, HDFS-10544.00.patch,
> HDFS-10544.01.patch, HDFS-10544.02.patch, HDFS-10544.03.patch,
> HDFS-10544.04.patch, HDFS-10544.05.patch
>
>
> Right now {{Balancer}} gets the NN URIs through
> {{DFSUtil#getNameServiceUris}}, which returns logical URIs in HA is enabled.
> If {{IPFailoverProxyProvider}} is used, {{Balancer}} will not be able to
> start.
> I think the bug is at {{DFSUtil#getNameServiceUris}}:
> {code}
> for (String nsId : getNameServiceIds(conf)) {
> if (HAUtil.isHAEnabled(conf, nsId)) {
> // Add the logical URI of the nameservice.
> try {
> ret.add(new URI(HdfsConstants.HDFS_URI_SCHEME + "://" + nsId));
> {code}
> Then {{if}} clause should also consider if the {{FailoverProxyProvider}} has
> {{useLogicalURI}} enabled. If not, {{getNameServiceUris}} should try to
> resolve the physical URI for this nsId.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10544) Balancer doesn't work with IPFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10544:
-
Affects Version/s: 2.6.1
> Balancer doesn't work with IPFailoverProxyProvider
> --
>
> Key: HDFS-10544
> URL: https://issues.apache.org/jira/browse/HDFS-10544
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, ha
>Affects Versions: 2.6.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.7.3, 2.9.0, 2.6.5, 3.0.0-alpha1
>
> Attachments: HDFS-10544-branch-2.7.patch, HDFS-10544.00.patch,
> HDFS-10544.01.patch, HDFS-10544.02.patch, HDFS-10544.03.patch,
> HDFS-10544.04.patch, HDFS-10544.05.patch
>
>
> Right now {{Balancer}} gets the NN URIs through
> {{DFSUtil#getNameServiceUris}}, which returns logical URIs in HA is enabled.
> If {{IPFailoverProxyProvider}} is used, {{Balancer}} will not be able to
> start.
> I think the bug is at {{DFSUtil#getNameServiceUris}}:
> {code}
> for (String nsId : getNameServiceIds(conf)) {
> if (HAUtil.isHAEnabled(conf, nsId)) {
> // Add the logical URI of the nameservice.
> try {
> ret.add(new URI(HdfsConstants.HDFS_URI_SCHEME + "://" + nsId));
> {code}
> Then {{if}} clause should also consider if the {{FailoverProxyProvider}} has
> {{useLogicalURI}} enabled. If not, {{getNameServiceUris}} should try to
> resolve the physical URI for this nsId.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10544) Balancer doesn't work with IPFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10544:
-
Target Version/s: 2.6.5 (was: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1)
> Balancer doesn't work with IPFailoverProxyProvider
> --
>
> Key: HDFS-10544
> URL: https://issues.apache.org/jira/browse/HDFS-10544
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, ha
>Affects Versions: 2.6.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.7.3, 2.9.0, 2.6.5, 3.0.0-alpha1
>
> Attachments: HDFS-10544-branch-2.7.patch, HDFS-10544.00.patch,
> HDFS-10544.01.patch, HDFS-10544.02.patch, HDFS-10544.03.patch,
> HDFS-10544.04.patch, HDFS-10544.05.patch
>
>
> Right now {{Balancer}} gets the NN URIs through
> {{DFSUtil#getNameServiceUris}}, which returns logical URIs in HA is enabled.
> If {{IPFailoverProxyProvider}} is used, {{Balancer}} will not be able to
> start.
> I think the bug is at {{DFSUtil#getNameServiceUris}}:
> {code}
> for (String nsId : getNameServiceIds(conf)) {
> if (HAUtil.isHAEnabled(conf, nsId)) {
> // Add the logical URI of the nameservice.
> try {
> ret.add(new URI(HdfsConstants.HDFS_URI_SCHEME + "://" + nsId));
> {code}
> Then {{if}} clause should also consider if the {{FailoverProxyProvider}} has
> {{useLogicalURI}} enabled. If not, {{getNameServiceUris}} should try to
> resolve the physical URI for this nsId.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10544) Balancer doesn't work with IPFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10544:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.6.5
2.7.3
Status: Resolved (was: Patch Available)
I just pushed to branch-2.7 and branch-2.6. Thanks again for the review from
[~shv]!
> Balancer doesn't work with IPFailoverProxyProvider
> --
>
> Key: HDFS-10544
> URL: https://issues.apache.org/jira/browse/HDFS-10544
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.7.3, 2.9.0, 2.6.5, 3.0.0-alpha1
>
> Attachments: HDFS-10544-branch-2.7.patch, HDFS-10544.00.patch,
> HDFS-10544.01.patch, HDFS-10544.02.patch, HDFS-10544.03.patch,
> HDFS-10544.04.patch, HDFS-10544.05.patch
>
>
> Right now {{Balancer}} gets the NN URIs through
> {{DFSUtil#getNameServiceUris}}, which returns logical URIs in HA is enabled.
> If {{IPFailoverProxyProvider}} is used, {{Balancer}} will not be able to
> start.
> I think the bug is at {{DFSUtil#getNameServiceUris}}:
> {code}
> for (String nsId : getNameServiceIds(conf)) {
> if (HAUtil.isHAEnabled(conf, nsId)) {
> // Add the logical URI of the nameservice.
> try {
> ret.add(new URI(HdfsConstants.HDFS_URI_SCHEME + "://" + nsId));
> {code}
> Then {{if}} clause should also consider if the {{FailoverProxyProvider}} has
> {{useLogicalURI}} enabled. If not, {{getNameServiceUris}} should try to
> resolve the physical URI for this nsId.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10544) Balancer doesn't work with IPFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10544:
-
Attachment: HDFS-10544-branch-2.7.patch
Attaching branch-2.7 patch to trigger Jenkins.
> Balancer doesn't work with IPFailoverProxyProvider
> --
>
> Key: HDFS-10544
> URL: https://issues.apache.org/jira/browse/HDFS-10544
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.9.0, 3.0.0-alpha1
>
> Attachments: HDFS-10544-branch-2.7.patch, HDFS-10544.00.patch,
> HDFS-10544.01.patch, HDFS-10544.02.patch, HDFS-10544.03.patch,
> HDFS-10544.04.patch, HDFS-10544.05.patch
>
>
> Right now {{Balancer}} gets the NN URIs through
> {{DFSUtil#getNameServiceUris}}, which returns logical URIs in HA is enabled.
> If {{IPFailoverProxyProvider}} is used, {{Balancer}} will not be able to
> start.
> I think the bug is at {{DFSUtil#getNameServiceUris}}:
> {code}
> for (String nsId : getNameServiceIds(conf)) {
> if (HAUtil.isHAEnabled(conf, nsId)) {
> // Add the logical URI of the nameservice.
> try {
> ret.add(new URI(HdfsConstants.HDFS_URI_SCHEME + "://" + nsId));
> {code}
> Then {{if}} clause should also consider if the {{FailoverProxyProvider}} has
> {{useLogicalURI}} enabled. If not, {{getNameServiceUris}} should try to
> resolve the physical URI for this nsId.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10544) Balancer doesn't work with IPFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10544:
-
Fix Version/s: 3.0.0-alpha1
2.9.0
2.8.0
> Balancer doesn't work with IPFailoverProxyProvider
> --
>
> Key: HDFS-10544
> URL: https://issues.apache.org/jira/browse/HDFS-10544
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.9.0, 3.0.0-alpha1
>
> Attachments: HDFS-10544.00.patch, HDFS-10544.01.patch,
> HDFS-10544.02.patch, HDFS-10544.03.patch, HDFS-10544.04.patch,
> HDFS-10544.05.patch
>
>
> Right now {{Balancer}} gets the NN URIs through
> {{DFSUtil#getNameServiceUris}}, which returns logical URIs in HA is enabled.
> If {{IPFailoverProxyProvider}} is used, {{Balancer}} will not be able to
> start.
> I think the bug is at {{DFSUtil#getNameServiceUris}}:
> {code}
> for (String nsId : getNameServiceIds(conf)) {
> if (HAUtil.isHAEnabled(conf, nsId)) {
> // Add the logical URI of the nameservice.
> try {
> ret.add(new URI(HdfsConstants.HDFS_URI_SCHEME + "://" + nsId));
> {code}
> Then {{if}} clause should also consider if the {{FailoverProxyProvider}} has
> {{useLogicalURI}} enabled. If not, {{getNameServiceUris}} should try to
> resolve the physical URI for this nsId.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10544) Balancer doesn't work with IPFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10544:
-
Attachment: HDFS-10544.05.patch
Thanks [~shv] for the review! Attaching v5 patch which fixes 2 checkstyle
issues in v4 (lines in {{TestDFSUtil}} too long). The other 2 checkstyle issues
are inherent with the original code style.
The reported test failures are unrelated and cannot be reproduced locally.
I'll commit v5 patch shortly.
> Balancer doesn't work with IPFailoverProxyProvider
> --
>
> Key: HDFS-10544
> URL: https://issues.apache.org/jira/browse/HDFS-10544
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-10544.00.patch, HDFS-10544.01.patch,
> HDFS-10544.02.patch, HDFS-10544.03.patch, HDFS-10544.04.patch,
> HDFS-10544.05.patch
>
>
> Right now {{Balancer}} gets the NN URIs through
> {{DFSUtil#getNameServiceUris}}, which returns logical URIs in HA is enabled.
> If {{IPFailoverProxyProvider}} is used, {{Balancer}} will not be able to
> start.
> I think the bug is at {{DFSUtil#getNameServiceUris}}:
> {code}
> for (String nsId : getNameServiceIds(conf)) {
> if (HAUtil.isHAEnabled(conf, nsId)) {
> // Add the logical URI of the nameservice.
> try {
> ret.add(new URI(HdfsConstants.HDFS_URI_SCHEME + "://" + nsId));
> {code}
> Then {{if}} clause should also consider if the {{FailoverProxyProvider}} has
> {{useLogicalURI}} enabled. If not, {{getNameServiceUris}} should try to
> resolve the physical URI for this nsId.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10544) Balancer doesn't work with IPFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10544:
-
Attachment: HDFS-10544.04.patch
Thanks [~shv] for the review! Updating the patch to address the comments.
bq. ... Not sure if this is what you expected.
I updated the test to set a config entry to map {{ns1}} to a physical URI. So
the {{else}} statement in {{DFSUtil#getNameServiceUris}} will be able to
resolve {{ns1}} to a physical URI. I think this is the correct configuration in
an environment that uses {{IPFailoverProxyProvider}}.
Also updated the comments. Please kindly review.
> Balancer doesn't work with IPFailoverProxyProvider
> --
>
> Key: HDFS-10544
> URL: https://issues.apache.org/jira/browse/HDFS-10544
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-10544.00.patch, HDFS-10544.01.patch,
> HDFS-10544.02.patch, HDFS-10544.03.patch, HDFS-10544.04.patch
>
>
> Right now {{Balancer}} gets the NN URIs through
> {{DFSUtil#getNameServiceUris}}, which returns logical URIs in HA is enabled.
> If {{IPFailoverProxyProvider}} is used, {{Balancer}} will not be able to
> start.
> I think the bug is at {{DFSUtil#getNameServiceUris}}:
> {code}
> for (String nsId : getNameServiceIds(conf)) {
> if (HAUtil.isHAEnabled(conf, nsId)) {
> // Add the logical URI of the nameservice.
> try {
> ret.add(new URI(HdfsConstants.HDFS_URI_SCHEME + "://" + nsId));
> {code}
> Then {{if}} clause should also consider if the {{FailoverProxyProvider}} has
> {{useLogicalURI}} enabled. If not, {{getNameServiceUris}} should try to
> resolve the physical URI for this nsId.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10544) Balancer doesn't work with IPFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10544:
-
Attachment: HDFS-10544.03.patch
Thought more about the two options. I think we should use the simpler approach
of throwing {{IllegalArgumentException}} before checking if the proxy provider
works with logical Uris. This is because no matter if the config uses
{{ConfiguredFailoverProxyProvider}} or {{IPFailoverProxyProvider}}, a legal URI
is needed by the constructor. So it's actually a valid requirement that
{{nsId}} can form a legal URI.
Also updating the unit test {{TestDFSUtil}}. Basically, 2 of the tests were
using logical URIs without specifying a {{FailoverProxyProvider}}. With such a
config, the correct behavior of {{DFSUtil#getInternalNameServices}} is to try
to resolve the logical URI, instead of waiting for the proxy provider to do so.
But in those 2 tests, the config doesn't have the correct
{{dfs.namenode.servicerpc-address}} entries to resolve the logical URIs.
> Balancer doesn't work with IPFailoverProxyProvider
> --
>
> Key: HDFS-10544
> URL: https://issues.apache.org/jira/browse/HDFS-10544
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-10544.00.patch, HDFS-10544.01.patch,
> HDFS-10544.02.patch, HDFS-10544.03.patch
>
>
> Right now {{Balancer}} gets the NN URIs through
> {{DFSUtil#getNameServiceUris}}, which returns logical URIs in HA is enabled.
> If {{IPFailoverProxyProvider}} is used, {{Balancer}} will not be able to
> start.
> I think the bug is at {{DFSUtil#getNameServiceUris}}:
> {code}
> for (String nsId : getNameServiceIds(conf)) {
> if (HAUtil.isHAEnabled(conf, nsId)) {
> // Add the logical URI of the nameservice.
> try {
> ret.add(new URI(HdfsConstants.HDFS_URI_SCHEME + "://" + nsId));
> {code}
> Then {{if}} clause should also consider if the {{FailoverProxyProvider}} has
> {{useLogicalURI}} enabled. If not, {{getNameServiceUris}} should try to
> resolve the physical URI for this nsId.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10544) Balancer doesn't work with IPFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10544:
-
Attachment: HDFS-10544.02.patch
Uploading v2 patch. So we have 2 alternatives:
* v1 patch: add {{useLogicalUri}} to take String as parameter
* v2 patch: throw {{IllegalArgumentException}} before trying to determine
whether to use logical URI.
I'm hesitating between the 2 options because the documentation is not
completely clear about {{dfs.nameservices}}. Could it be an arbitrary string?
If that arbitrary string fails to form a legal URI (when prefixed with
{{hdfs://}}), shall we treat it just as a Runtime exception, or we should look
for that arbitrary string prefixed with {{dfs.namenode.servicerpc-address}} in
the config?
Any suggestions are very welcome.
> Balancer doesn't work with IPFailoverProxyProvider
> --
>
> Key: HDFS-10544
> URL: https://issues.apache.org/jira/browse/HDFS-10544
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-10544.00.patch, HDFS-10544.01.patch,
> HDFS-10544.02.patch
>
>
> Right now {{Balancer}} gets the NN URIs through
> {{DFSUtil#getNameServiceUris}}, which returns logical URIs in HA is enabled.
> If {{IPFailoverProxyProvider}} is used, {{Balancer}} will not be able to
> start.
> I think the bug is at {{DFSUtil#getNameServiceUris}}:
> {code}
> for (String nsId : getNameServiceIds(conf)) {
> if (HAUtil.isHAEnabled(conf, nsId)) {
> // Add the logical URI of the nameservice.
> try {
> ret.add(new URI(HdfsConstants.HDFS_URI_SCHEME + "://" + nsId));
> {code}
> Then {{if}} clause should also consider if the {{FailoverProxyProvider}} has
> {{useLogicalURI}} enabled. If not, {{getNameServiceUris}} should try to
> resolve the physical URI for this nsId.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10544) Balancer doesn't work with IPFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10544:
-
Attachment: HDFS-10544.01.patch
v0 patch didn't work because {{HAUtil#useLogicalUri}} expects an URI rather
than String (btw sorry I missed the build failure before uploading patch). I
don't think this is a good API design, because nameservice ID can be an
arbitrary string.
But to avoid changing public API {{HAUtil#useLogicalUri}}, in v1 patch I added
another API to get the boolean condition from string. Also using the static
method to check, rather than creating an instance of the proxy provider.
> Balancer doesn't work with IPFailoverProxyProvider
> --
>
> Key: HDFS-10544
> URL: https://issues.apache.org/jira/browse/HDFS-10544
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-10544.00.patch, HDFS-10544.01.patch
>
>
> Right now {{Balancer}} gets the NN URIs through
> {{DFSUtil#getNameServiceUris}}, which returns logical URIs in HA is enabled.
> If {{IPFailoverProxyProvider}} is used, {{Balancer}} will not be able to
> start.
> I think the bug is at {{DFSUtil#getNameServiceUris}}:
> {code}
> for (String nsId : getNameServiceIds(conf)) {
> if (HAUtil.isHAEnabled(conf, nsId)) {
> // Add the logical URI of the nameservice.
> try {
> ret.add(new URI(HdfsConstants.HDFS_URI_SCHEME + "://" + nsId));
> {code}
> Then {{if}} clause should also consider if the {{FailoverProxyProvider}} has
> {{useLogicalURI}} enabled. If not, {{getNameServiceUris}} should try to
> resolve the physical URI for this nsId.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10544) Balancer doesn't work with IPFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10544:
-
Attachment: HDFS-10544.00.patch
Submitting initial patch.
{{HAUtil#useLogicalUri}} can potentially be optimized to avoid creating an
instance of {{AbstractNNFailoverProxyProvider}}.
> Balancer doesn't work with IPFailoverProxyProvider
> --
>
> Key: HDFS-10544
> URL: https://issues.apache.org/jira/browse/HDFS-10544
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-10544.00.patch
>
>
> Right now {{Balancer}} gets the NN URIs through
> {{DFSUtil#getNameServiceUris}}, which returns logical URIs in HA is enabled.
> If {{IPFailoverProxyProvider}} is used, {{Balancer}} will not be able to
> start.
> I think the bug is at {{DFSUtil#getNameServiceUris}}:
> {code}
> for (String nsId : getNameServiceIds(conf)) {
> if (HAUtil.isHAEnabled(conf, nsId)) {
> // Add the logical URI of the nameservice.
> try {
> ret.add(new URI(HdfsConstants.HDFS_URI_SCHEME + "://" + nsId));
> {code}
> Then {{if}} clause should also consider if the {{FailoverProxyProvider}} has
> {{useLogicalURI}} enabled. If not, {{getNameServiceUris}} should try to
> resolve the physical URI for this nsId.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10544) Balancer doesn't work with IPFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10544:
-
Status: Patch Available (was: Open)
> Balancer doesn't work with IPFailoverProxyProvider
> --
>
> Key: HDFS-10544
> URL: https://issues.apache.org/jira/browse/HDFS-10544
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> Right now {{Balancer}} gets the NN URIs through
> {{DFSUtil#getNameServiceUris}}, which returns logical URIs in HA is enabled.
> If {{IPFailoverProxyProvider}} is used, {{Balancer}} will not be able to
> start.
> I think the bug is at {{DFSUtil#getNameServiceUris}}:
> {code}
> for (String nsId : getNameServiceIds(conf)) {
> if (HAUtil.isHAEnabled(conf, nsId)) {
> // Add the logical URI of the nameservice.
> try {
> ret.add(new URI(HdfsConstants.HDFS_URI_SCHEME + "://" + nsId));
> {code}
> Then {{if}} clause should also consider if the {{FailoverProxyProvider}} has
> {{useLogicalURI}} enabled. If not, {{getNameServiceUris}} should try to
> resolve the physical URI for this nsId.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
