[jira] [Commented] (HDFS-15294) RBF: Balance data across federation namespaces with DistCp and snapshot diff
[
https://issues.apache.org/jira/browse/HDFS-15294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17103114#comment-17103114
]
Jinglun commented on HDFS-15294:
Hi [~linyiqun], thanks your comments, I agree with your opinion. HDFS-15087
only works under very strict conditions with many restrictions and in user's
perspective it's not very efficient. The distcp diff copy way is much better.
I think I should close the HDFS-15087 and move all the sub-tasks under this
Jira.
> RBF: Balance data across federation namespaces with DistCp and snapshot diff
>
>
> Key: HDFS-15294
> URL: https://issues.apache.org/jira/browse/HDFS-15294
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: BalanceProcedureScheduler.png, HDFS-15294.001.patch,
> HDFS-15294.002.patch, HDFS-15294.003.patch, HDFS-15294.003.reupload.patch,
> HDFS-15294.004.patch, HDFS-15294.005.patch, HDFS-15294.006.patch,
> HDFS-15294.007.patch, distcp-balance.pdf, distcp-balance.v2.pdf
>
>
> This jira introduces a new balance command 'fedbalance' that is ran by the
> administrator. The process is:
> 1. Use distcp and snapshot diff to sync data between src and dst until they
> are the same.
> 2. Update mount table in Router.
> 3. Delete the src to trash.
>
> The patch is too big to review, so I split it into 2 patches:
> Phase 1 / The State Machine(BalanceProcedureScheduler): Including the
> abstraction of job and scheduler model.
> {code:java}
> org.apache.hadoop.hdfs.procedure.BalanceProcedureScheduler;
> org.apache.hadoop.hdfs.procedure.BalanceProcedureConfigKeys;
> org.apache.hadoop.hdfs.procedure.BalanceProcedure;
> org.apache.hadoop.hdfs.procedure.BalanceJob;
> org.apache.hadoop.hdfs.procedure.BalanceJournal;
> org.apache.hadoop.hdfs.procedure.HDFSJournal;
> {code}
> Phase 2 / The DistCpFedBalance: It's an implementation of BalanceJob. HDFS-15346>
> {code:java}
> org.apache.hadoop.hdfs.server.federation.procedure.MountTableProcedure;
> org.apache.hadoop.tools.DistCpFedBalance;
> org.apache.hadoop.tools.DistCpProcedure;
> org.apache.hadoop.tools.FedBalance;
> org.apache.hadoop.tools.FedBalanceConfigs;
> org.apache.hadoop.tools.FedBalanceContext;
> org.apache.hadoop.tools.TrashProcedure;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15294) RBF: Balance data across federation namespaces with DistCp and snapshot diff
[
https://issues.apache.org/jira/browse/HDFS-15294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17103104#comment-17103104
]
Yiqun Lin commented on HDFS-15294:
--
Hi [~LiJinglun], I agree that starting with distcp diff copy is a good choice
and be more acceptable.
{quote}I didn't give up the HDFS-15087 way. Though it impacts user a lot but is
more efficient.
{quote}
>From my personal opinion, user impact is more important than efficiency. The
>efficiency of fedbalace job is impacted by multiple diff distcp be scheduled
>not the balance data slowness. So it's not a big problem I think. Any thoughts
>from others?
{quote}Another question is all the suggestions seem to be related to the
DistCpFedBalance part, which is in the second patch. Could you help to review
the first patch so I can start working on the second one.
{quote}
Yes, I can help do this part of review work in recent days.
> RBF: Balance data across federation namespaces with DistCp and snapshot diff
>
>
> Key: HDFS-15294
> URL: https://issues.apache.org/jira/browse/HDFS-15294
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: BalanceProcedureScheduler.png, HDFS-15294.001.patch,
> HDFS-15294.002.patch, HDFS-15294.003.patch, HDFS-15294.003.reupload.patch,
> HDFS-15294.004.patch, HDFS-15294.005.patch, HDFS-15294.006.patch,
> HDFS-15294.007.patch, distcp-balance.pdf, distcp-balance.v2.pdf
>
>
> This jira introduces a new balance command 'fedbalance' that is ran by the
> administrator. The process is:
> 1. Use distcp and snapshot diff to sync data between src and dst until they
> are the same.
> 2. Update mount table in Router.
> 3. Delete the src to trash.
>
> The patch is too big to review, so I split it into 2 patches:
> Phase 1 / The State Machine(BalanceProcedureScheduler): Including the
> abstraction of job and scheduler model.
> {code:java}
> org.apache.hadoop.hdfs.procedure.BalanceProcedureScheduler;
> org.apache.hadoop.hdfs.procedure.BalanceProcedureConfigKeys;
> org.apache.hadoop.hdfs.procedure.BalanceProcedure;
> org.apache.hadoop.hdfs.procedure.BalanceJob;
> org.apache.hadoop.hdfs.procedure.BalanceJournal;
> org.apache.hadoop.hdfs.procedure.HDFSJournal;
> {code}
> Phase 2 / The DistCpFedBalance: It's an implementation of BalanceJob. HDFS-15346>
> {code:java}
> org.apache.hadoop.hdfs.server.federation.procedure.MountTableProcedure;
> org.apache.hadoop.tools.DistCpFedBalance;
> org.apache.hadoop.tools.DistCpProcedure;
> org.apache.hadoop.tools.FedBalance;
> org.apache.hadoop.tools.FedBalanceConfigs;
> org.apache.hadoop.tools.FedBalanceContext;
> org.apache.hadoop.tools.TrashProcedure;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15346) RBF: Balance data across federation namespaces with DistCp and snapshot diff / Step 2: The DistCpFedBalance.
[ https://issues.apache.org/jira/browse/HDFS-15346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinglun reassigned HDFS-15346: -- Assignee: Jinglun > RBF: Balance data across federation namespaces with DistCp and snapshot diff > / Step 2: The DistCpFedBalance. > > > Key: HDFS-15346 > URL: https://issues.apache.org/jira/browse/HDFS-15346 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > > Patch in HDFS-15294 is too big to review so we split it into 2 patches. This > is the second one. Detail can be found at HDFS-15294. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15346) RBF: Balance data across federation namespaces with DistCp and snapshot diff / Step 2: The DistCpFedBalance.
[ https://issues.apache.org/jira/browse/HDFS-15346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinglun updated HDFS-15346: --- Description: Patch in HDFS-15294 is too big to review so we split it into 2 patches. This is the second one. Detail can be found at HDFS-15294. > RBF: Balance data across federation namespaces with DistCp and snapshot diff > / Step 2: The DistCpFedBalance. > > > Key: HDFS-15346 > URL: https://issues.apache.org/jira/browse/HDFS-15346 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jinglun >Priority: Major > > Patch in HDFS-15294 is too big to review so we split it into 2 patches. This > is the second one. Detail can be found at HDFS-15294. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15346) RBF: Balance data across federation namespaces with DistCp and snapshot diff / Step 2: The DistCpFedBalance.
Jinglun created HDFS-15346: -- Summary: RBF: Balance data across federation namespaces with DistCp and snapshot diff / Step 2: The DistCpFedBalance. Key: HDFS-15346 URL: https://issues.apache.org/jira/browse/HDFS-15346 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jinglun -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15294) RBF: Balance data across federation namespaces with DistCp and snapshot diff
[
https://issues.apache.org/jira/browse/HDFS-15294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jinglun updated HDFS-15294:
---
Description:
This jira introduces a new balance command 'fedbalance' that is ran by the
administrator. The process is:
1. Use distcp and snapshot diff to sync data between src and dst until they
are the same.
2. Update mount table in Router.
3. Delete the src to trash.
The patch is too big to review, so I split it into 2 patches:
Phase 1 / The State Machine(BalanceProcedureScheduler): Including the
abstraction of job and scheduler model.
{code:java}
org.apache.hadoop.hdfs.procedure.BalanceProcedureScheduler;
org.apache.hadoop.hdfs.procedure.BalanceProcedureConfigKeys;
org.apache.hadoop.hdfs.procedure.BalanceProcedure;
org.apache.hadoop.hdfs.procedure.BalanceJob;
org.apache.hadoop.hdfs.procedure.BalanceJournal;
org.apache.hadoop.hdfs.procedure.HDFSJournal;
{code}
Phase 2 / The DistCpFedBalance: It's an implementation of BalanceJob.
{code:java}
org.apache.hadoop.hdfs.server.federation.procedure.MountTableProcedure;
org.apache.hadoop.tools.DistCpFedBalance;
org.apache.hadoop.tools.DistCpProcedure;
org.apache.hadoop.tools.FedBalance;
org.apache.hadoop.tools.FedBalanceConfigs;
org.apache.hadoop.tools.FedBalanceContext;
org.apache.hadoop.tools.TrashProcedure;
{code}
was:
This jira introduces a new balance command 'fedbalance' that is ran by the
administrator. The process is:
1. Use distcp and snapshot diff to sync data between src and dst until they
are the same.
2. Update mount table in Router.
3. Delete the src to trash.
The patch is too big to review, so I split it into 2 patches:
Phase 1 / The State Machine(BalanceProcedureScheduler): Including the
abstraction of job and scheduler model.
{code:java}
org.apache.hadoop.hdfs.procedure.BalanceProcedureScheduler;
org.apache.hadoop.hdfs.procedure.BalanceProcedureConfigKeys;
org.apache.hadoop.hdfs.procedure.BalanceProcedure;
org.apache.hadoop.hdfs.procedure.BalanceJob;
org.apache.hadoop.hdfs.procedure.BalanceJournal;
org.apache.hadoop.hdfs.procedure.HDFSJournal;
{code}
Phase 2 / The DistCpFedBalance: It's an implementation of BalanceJob.
{code:java}
org.apache.hadoop.hdfs.server.federation.procedure.MountTableProcedure;
org.apache.hadoop.tools.DistCpFedBalance;
org.apache.hadoop.tools.DistCpProcedure;
org.apache.hadoop.tools.FedBalance;
org.apache.hadoop.tools.FedBalanceConfigs;
org.apache.hadoop.tools.FedBalanceContext;
org.apache.hadoop.tools.TrashProcedure;
{code}
> RBF: Balance data across federation namespaces with DistCp and snapshot diff
>
>
> Key: HDFS-15294
> URL: https://issues.apache.org/jira/browse/HDFS-15294
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: BalanceProcedureScheduler.png, HDFS-15294.001.patch,
> HDFS-15294.002.patch, HDFS-15294.003.patch, HDFS-15294.003.reupload.patch,
> HDFS-15294.004.patch, HDFS-15294.005.patch, HDFS-15294.006.patch,
> HDFS-15294.007.patch, distcp-balance.pdf, distcp-balance.v2.pdf
>
>
> This jira introduces a new balance command 'fedbalance' that is ran by the
> administrator. The process is:
> 1. Use distcp and snapshot diff to sync data between src and dst until they
> are the same.
> 2. Update mount table in Router.
> 3. Delete the src to trash.
>
> The patch is too big to review, so I split it into 2 patches:
> Phase 1 / The State Machine(BalanceProcedureScheduler): Including the
> abstraction of job and scheduler model.
> {code:java}
> org.apache.hadoop.hdfs.procedure.BalanceProcedureScheduler;
> org.apache.hadoop.hdfs.procedure.BalanceProcedureConfigKeys;
> org.apache.hadoop.hdfs.procedure.BalanceProcedure;
> org.apache.hadoop.hdfs.procedure.BalanceJob;
> org.apache.hadoop.hdfs.procedure.BalanceJournal;
> org.apache.hadoop.hdfs.procedure.HDFSJournal;
> {code}
> Phase 2 / The DistCpFedBalance: It's an implementation of BalanceJob. HDFS-15346>
> {code:java}
> org.apache.hadoop.hdfs.server.federation.procedure.MountTableProcedure;
> org.apache.hadoop.tools.DistCpFedBalance;
> org.apache.hadoop.tools.DistCpProcedure;
> org.apache.hadoop.tools.FedBalance;
> org.apache.hadoop.tools.FedBalanceConfigs;
> org.apache.hadoop.tools.FedBalanceContext;
> org.apache.hadoop.tools.TrashProcedure;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15294) RBF: Balance data across federation namespaces with DistCp and snapshot diff
[
https://issues.apache.org/jira/browse/HDFS-15294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17103079#comment-17103079
]
Jinglun commented on HDFS-15294:
Hi [~linyiqun], thanks your nice comments.
{quote}Current fedbalance tool seems can be only used for the case of RBF, but
actually this tool can also help balance data in normal federation case. It
will be better to additionally add an option to allow submit BalancerJob
without MountTableProcedure.
{quote}
Yes you are right. I'll update the next patch with an option for normal
federation case.
{quote}Now current implementation completely different with the origin design
in HDFS-15087 which uses the way like, block writes -> saveTree -> graftTree ->
hardlink -> update router mount table. Does that mean you change the way that
you implemented before? But anyway, current fedbalance tool way looks good, it
doesn't lock the folder and can minimum the user impact. [~LiJinglun], if you
already have a new design based on fedbalance way, please update the new design
doc. Thanks.
{quote}
The balance in RBF is a big work. I should split it into many sub-tasks. I
think staring it with distcp diff is a good choice. I didn't give up the
HDFS-15087 way. Though it impacts user a lot but is more efficient. But
starting with it is not a good idea as it is not universal and covers only a
little scenarios. In the future might be we can use it for fast rename small
paths across federation sub-clusters.
The distcp-balance.v2.pdf is the design doc for this jira. May be I should use
a google doc for this. So all the suggestions can be updated in time.
Another question is all the suggestions seem to be related to the
DistCpFedBalance part, which is in the second patch. Could you help to review
the first patch so I can start working on the second one. Thanks very much :D !
> RBF: Balance data across federation namespaces with DistCp and snapshot diff
>
>
> Key: HDFS-15294
> URL: https://issues.apache.org/jira/browse/HDFS-15294
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: BalanceProcedureScheduler.png, HDFS-15294.001.patch,
> HDFS-15294.002.patch, HDFS-15294.003.patch, HDFS-15294.003.reupload.patch,
> HDFS-15294.004.patch, HDFS-15294.005.patch, HDFS-15294.006.patch,
> HDFS-15294.007.patch, distcp-balance.pdf, distcp-balance.v2.pdf
>
>
> This jira introduces a new balance command 'fedbalance' that is ran by the
> administrator. The process is:
> 1. Use distcp and snapshot diff to sync data between src and dst until they
> are the same.
> 2. Update mount table in Router.
> 3. Delete the src to trash.
>
> The patch is too big to review, so I split it into 2 patches:
> Phase 1 / The State Machine(BalanceProcedureScheduler): Including the
> abstraction of job and scheduler model.
> {code:java}
> org.apache.hadoop.hdfs.procedure.BalanceProcedureScheduler;
> org.apache.hadoop.hdfs.procedure.BalanceProcedureConfigKeys;
> org.apache.hadoop.hdfs.procedure.BalanceProcedure;
> org.apache.hadoop.hdfs.procedure.BalanceJob;
> org.apache.hadoop.hdfs.procedure.BalanceJournal;
> org.apache.hadoop.hdfs.procedure.HDFSJournal;
> {code}
> Phase 2 / The DistCpFedBalance: It's an implementation of BalanceJob.
>
> {code:java}
> org.apache.hadoop.hdfs.server.federation.procedure.MountTableProcedure;
> org.apache.hadoop.tools.DistCpFedBalance;
> org.apache.hadoop.tools.DistCpProcedure;
> org.apache.hadoop.tools.FedBalance;
> org.apache.hadoop.tools.FedBalanceConfigs;
> org.apache.hadoop.tools.FedBalanceContext;
> org.apache.hadoop.tools.TrashProcedure;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15306) Make mount-table to read from central place ( Let's say from HDFS)
[ https://issues.apache.org/jira/browse/HDFS-15306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17103059#comment-17103059 ] Uma Maheswara Rao G edited comment on HDFS-15306 at 5/9/20, 2:27 AM: - I added the PR. This PR provides the flexibility to load the remote mount-table configuration when file available in hadoop compatible filesystems. was (Author: umamaheswararao): I updated the PR with idea of this patch. This patch provides the flexibility to load the remote mount-table configuration when file available in hadoop compatible filesystems. > Make mount-table to read from central place ( Let's say from HDFS) > -- > > Key: HDFS-15306 > URL: https://issues.apache.org/jira/browse/HDFS-15306 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: configuration, hadoop-client >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > ViewFsOverloadScheme should be able to read mount-table.xml configuration > from remote servers. > Below are the discussed options in design doc: > # XInclude and HTTP Server ( including WebHDFS) > # Hadoop Compatible FS (*HCFS)* > a) Keep mount-table in Hadoop compatible FS > b)Read mount-table from Hadoop compatible FS using Xinclude > We prefer to have 1 and 2a. For 1 we don't need to modify any code. So, this > Jira can cover 2a. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15306) Make mount-table to read from central place ( Let's say from HDFS)
[ https://issues.apache.org/jira/browse/HDFS-15306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15306: --- Status: Patch Available (was: Open) I updated the PR with idea of this patch. This patch provides the flexibility to load the remote mount-table configuration when file available in hadoop compatible filesystems. > Make mount-table to read from central place ( Let's say from HDFS) > -- > > Key: HDFS-15306 > URL: https://issues.apache.org/jira/browse/HDFS-15306 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: configuration, hadoop-client >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > ViewFsOverloadScheme should be able to read mount-table.xml configuration > from remote servers. > Below are the discussed options in design doc: > # XInclude and HTTP Server ( including WebHDFS) > # Hadoop Compatible FS (*HCFS)* > a) Keep mount-table in Hadoop compatible FS > b)Read mount-table from Hadoop compatible FS using Xinclude > We prefer to have 1 and 2a. For 1 we don't need to modify any code. So, this > Jira can cover 2a. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15345) RBF: RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442
[ https://issues.apache.org/jira/browse/HDFS-15345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-15345: --- Summary: RBF: RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442 (was: RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442) > RBF: RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups > after HADOOP-13442 > > > Key: HDFS-15345 > URL: https://issues.apache.org/jira/browse/HDFS-15345 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.5 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > > HADOOP-13442 added UGI#getGroups to avoid list->array->list conversions. This > ticket is opened to change RouterPermissionChecker#checkSuperuserPrivilege > to use UGI#getGroups. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14999) Avoid Potential Infinite Loop in DFSNetworkTopology
[
https://issues.apache.org/jira/browse/HDFS-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102860#comment-17102860
]
Ayush Saxena commented on HDFS-14999:
-
Thanx [~vinayakumarb] for the review.
I ran the said test. The performance isn’t getting affected. Both took similar
times in multiple runs, with the newer one taking negligible number of
milliseconds less.
This shouldn’t impact performance ideally.
Please review!!’
> Avoid Potential Infinite Loop in DFSNetworkTopology
> ---
>
> Key: HDFS-14999
> URL: https://issues.apache.org/jira/browse/HDFS-14999
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14999-01.patch
>
>
> {code:java}
> do {
> chosen = chooseRandomWithStorageTypeAndExcludeRoot(root, excludeRoot,
> type);
> if (excludedNodes == null || !excludedNodes.contains(chosen)) {
> break;
> } else {
> LOG.debug("Node {} is excluded, continuing.", chosen);
> }
> } while (true);
> {code}
> Observed this loop getting stuck as part of testing HDFS-14913.
> There should be some exit condition or max retries here
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15345) RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442
Xiaoyu Yao created HDFS-15345: - Summary: RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442 Key: HDFS-15345 URL: https://issues.apache.org/jira/browse/HDFS-15345 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.7.5 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao HADOOP-13442 added UGI#getGroups to avoid list->array->list conversions. This ticket is opened to change DataNode#checkSuperuserPrivilege to use UGI#getGroups. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15345) RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442
[ https://issues.apache.org/jira/browse/HDFS-15345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-15345: -- Description: HADOOP-13442 added UGI#getGroups to avoid list->array->list conversions. This ticket is opened to change RouterPermissionChecker#checkSuperuserPrivilege to use UGI#getGroups. (was: HADOOP-13442 added UGI#getGroups to avoid list->array->list conversions. This ticket is opened to change DataNode#checkSuperuserPrivilege to use UGI#getGroups. ) > RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups > after HADOOP-13442 > > > Key: HDFS-15345 > URL: https://issues.apache.org/jira/browse/HDFS-15345 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.5 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > > HADOOP-13442 added UGI#getGroups to avoid list->array->list conversions. This > ticket is opened to change RouterPermissionChecker#checkSuperuserPrivilege > to use UGI#getGroups. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15344) DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442
[ https://issues.apache.org/jira/browse/HDFS-15344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102786#comment-17102786 ] Xiaoyu Yao commented on HDFS-15344: --- Also find a similar issue in RouterPermissionChecker#checkSuperuserPrivilege. It could be fixed in the same patch but decide to open a separate so that we can have a clean backport to branch-2 without router feature in future. > DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442 > > > Key: HDFS-15344 > URL: https://issues.apache.org/jira/browse/HDFS-15344 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.5 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > > HADOOP-13442 added UGI#getGroups to avoid list->array->list conversions. This > ticket is opened to change DataNode#checkSuperuserPrivilege to use > UGI#getGroups. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15344) DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442
[ https://issues.apache.org/jira/browse/HDFS-15344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-15344: -- Status: Patch Available (was: Open) > DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442 > > > Key: HDFS-15344 > URL: https://issues.apache.org/jira/browse/HDFS-15344 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.5 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > > HADOOP-13442 added UGI#getGroups to avoid list->array->list conversions. This > ticket is opened to change DataNode#checkSuperuserPrivilege to use > UGI#getGroups. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15344) DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442
Xiaoyu Yao created HDFS-15344: - Summary: DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442 Key: HDFS-15344 URL: https://issues.apache.org/jira/browse/HDFS-15344 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.7.5 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao HADOOP-13442 added UGI#getGroups to avoid list->array->list conversions. This ticket is opened to change DataNode#checkSuperuserPrivilege to use UGI#getGroups. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15294) RBF: Balance data across federation namespaces with DistCp and snapshot diff
[
https://issues.apache.org/jira/browse/HDFS-15294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102677#comment-17102677
]
Yiqun Lin edited comment on HDFS-15294 at 5/8/20, 3:35 PM:
---
Hi [~LiJinglun], thanks for the great work on this!
Some comments for current design:
* Current fedbalance tool seems can be only used for the case of RBF, but
actually this tool can also help balance data in normal federation case. It
will be better to additionally add an option to allow submit BalancerJob
without MountTableProcedure.
* Now current implementation completely different with the origin design in
HDFS-15087 which uses the way like, block writes -> saveTree -> graftTree ->
hardlink -> update router mount table. Does that mean you change the way that
you implemented before? But anyway, current fedbalance tool way looks good, it
doesn't lock the folder and can minimum the user impact. [~LiJinglun], if you
already have a new design based on fedbalance way, please update the new design
doc. Thanks.
was (Author: linyiqun):
Hi [~LiJinglun], thanks for the great work on this!
Some comments for current design:
* Current fedbalance tool seems can be only used for the case of RBF, but
actually this tool can also help balance data in normal federation case. It
will be better to additionally add an option to allow submit BalancerJob
without MountTableProcedure.
* Now current implementation completely different with the origin design in
HDFS-15087 which uses the way like, block writes -> saveTree -> graftTree ->
hardlink -> update router mount table. Does that mean you change the way that
you implemented before? But anyway, current fedbalance tool way looks good, it
doesn't lock the folder and can minimum the user impact.
> RBF: Balance data across federation namespaces with DistCp and snapshot diff
>
>
> Key: HDFS-15294
> URL: https://issues.apache.org/jira/browse/HDFS-15294
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: BalanceProcedureScheduler.png, HDFS-15294.001.patch,
> HDFS-15294.002.patch, HDFS-15294.003.patch, HDFS-15294.003.reupload.patch,
> HDFS-15294.004.patch, HDFS-15294.005.patch, HDFS-15294.006.patch,
> HDFS-15294.007.patch, distcp-balance.pdf, distcp-balance.v2.pdf
>
>
> This jira introduces a new balance command 'fedbalance' that is ran by the
> administrator. The process is:
> 1. Use distcp and snapshot diff to sync data between src and dst until they
> are the same.
> 2. Update mount table in Router.
> 3. Delete the src to trash.
>
> The patch is too big to review, so I split it into 2 patches:
> Phase 1 / The State Machine(BalanceProcedureScheduler): Including the
> abstraction of job and scheduler model.
> {code:java}
> org.apache.hadoop.hdfs.procedure.BalanceProcedureScheduler;
> org.apache.hadoop.hdfs.procedure.BalanceProcedureConfigKeys;
> org.apache.hadoop.hdfs.procedure.BalanceProcedure;
> org.apache.hadoop.hdfs.procedure.BalanceJob;
> org.apache.hadoop.hdfs.procedure.BalanceJournal;
> org.apache.hadoop.hdfs.procedure.HDFSJournal;
> {code}
> Phase 2 / The DistCpFedBalance: It's an implementation of BalanceJob.
>
> {code:java}
> org.apache.hadoop.hdfs.server.federation.procedure.MountTableProcedure;
> org.apache.hadoop.tools.DistCpFedBalance;
> org.apache.hadoop.tools.DistCpProcedure;
> org.apache.hadoop.tools.FedBalance;
> org.apache.hadoop.tools.FedBalanceConfigs;
> org.apache.hadoop.tools.FedBalanceContext;
> org.apache.hadoop.tools.TrashProcedure;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15294) RBF: Balance data across federation namespaces with DistCp and snapshot diff
[
https://issues.apache.org/jira/browse/HDFS-15294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102677#comment-17102677
]
Yiqun Lin commented on HDFS-15294:
--
Hi [~LiJinglun], thanks for the great work on this!
Some comments for current design:
* Current fedbalance tool seems can be only used for the case of RBF, but
actually this tool can also help balance data in normal federation case. It
will be better to additionally add an option to allow submit BalancerJob
without MountTableProcedure.
* Now current implementation completely different with the origin design in
HDFS-15087 which uses the way like, block writes -> saveTree -> graftTree ->
hardlink -> update router mount table. Does that mean you change the way that
you implemented before? But anyway, current fedbalance tool way looks good, it
doesn't lock the folder and can minimum the user impact.
> RBF: Balance data across federation namespaces with DistCp and snapshot diff
>
>
> Key: HDFS-15294
> URL: https://issues.apache.org/jira/browse/HDFS-15294
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: BalanceProcedureScheduler.png, HDFS-15294.001.patch,
> HDFS-15294.002.patch, HDFS-15294.003.patch, HDFS-15294.003.reupload.patch,
> HDFS-15294.004.patch, HDFS-15294.005.patch, HDFS-15294.006.patch,
> HDFS-15294.007.patch, distcp-balance.pdf, distcp-balance.v2.pdf
>
>
> This jira introduces a new balance command 'fedbalance' that is ran by the
> administrator. The process is:
> 1. Use distcp and snapshot diff to sync data between src and dst until they
> are the same.
> 2. Update mount table in Router.
> 3. Delete the src to trash.
>
> The patch is too big to review, so I split it into 2 patches:
> Phase 1 / The State Machine(BalanceProcedureScheduler): Including the
> abstraction of job and scheduler model.
> {code:java}
> org.apache.hadoop.hdfs.procedure.BalanceProcedureScheduler;
> org.apache.hadoop.hdfs.procedure.BalanceProcedureConfigKeys;
> org.apache.hadoop.hdfs.procedure.BalanceProcedure;
> org.apache.hadoop.hdfs.procedure.BalanceJob;
> org.apache.hadoop.hdfs.procedure.BalanceJournal;
> org.apache.hadoop.hdfs.procedure.HDFSJournal;
> {code}
> Phase 2 / The DistCpFedBalance: It's an implementation of BalanceJob.
>
> {code:java}
> org.apache.hadoop.hdfs.server.federation.procedure.MountTableProcedure;
> org.apache.hadoop.tools.DistCpFedBalance;
> org.apache.hadoop.tools.DistCpProcedure;
> org.apache.hadoop.tools.FedBalance;
> org.apache.hadoop.tools.FedBalanceConfigs;
> org.apache.hadoop.tools.FedBalanceContext;
> org.apache.hadoop.tools.TrashProcedure;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14999) Avoid Potential Infinite Loop in DFSNetworkTopology
[
https://issues.apache.org/jira/browse/HDFS-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102653#comment-17102653
]
Vinayakumar B commented on HDFS-14999:
--
changes looks fine to me.
It would be better to have a benchmark done on this.
Perhaps the you can check \{{TestDFSNetworkTopologyPerformance}} class for
changes with excluded nodes.
Results before and after patch would help.
> Avoid Potential Infinite Loop in DFSNetworkTopology
> ---
>
> Key: HDFS-14999
> URL: https://issues.apache.org/jira/browse/HDFS-14999
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14999-01.patch
>
>
> {code:java}
> do {
> chosen = chooseRandomWithStorageTypeAndExcludeRoot(root, excludeRoot,
> type);
> if (excludedNodes == null || !excludedNodes.contains(chosen)) {
> break;
> } else {
> LOG.debug("Node {} is excluded, continuing.", chosen);
> }
> } while (true);
> {code}
> Observed this loop getting stuck as part of testing HDFS-14913.
> There should be some exit condition or max retries here
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15270) Account for *env == NULL in hdfsThreadDestructor
[ https://issues.apache.org/jira/browse/HDFS-15270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102609#comment-17102609 ] Babneet Singh commented on HDFS-15270: -- Requested issue's originator to respond regarding cherrypick/backport: https://github.com/eclipse/openj9/issues/7752#issuecomment-625826505. > Account for *env == NULL in hdfsThreadDestructor > > > Key: HDFS-15270 > URL: https://issues.apache.org/jira/browse/HDFS-15270 > Project: Hadoop HDFS > Issue Type: Bug > Environment: Please refer to the "steps to reproduce" the failure in > https://github.com/eclipse/openj9/issues/7752#issue-521732953. >Reporter: Babneet Singh >Assignee: Babneet Singh >Priority: Major > Fix For: 3.4.0 > > > OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is > invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread > termination, accessing JNIEnv in hdfsThreadDestructor (*env)->GetJavaVM, > yields a SIGSEGV since *env is NULL after thread cleanup is performed. > The main purpose of hdfsThreadDestructor is to invoke > DetachCurrentThread, which performs thread cleanup in OpenJ9. Since > OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked, > hdfsThreadDestructor should account for *env == NULL and skip > DetachCurrentThread. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15341) HDFS Move viewfs path to trash failed.
[
https://issues.apache.org/jira/browse/HDFS-15341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102541#comment-17102541
]
Vince.Feng commented on HDFS-15341:
---
@[acmurthy|https://github.com/acmurthy] @[aajisaka|https://github.com/aajisaka]
@[xiaoyuyao|https://github.com/xiaoyuyao] @[arp7|https://github.com/arp7] @[
acmurthy|https://github.com/acmurthy]
> HDFS Move viewfs path to trash failed.
> --
>
> Key: HDFS-15341
> URL: https://issues.apache.org/jira/browse/HDFS-15341
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.10.0, 3.3.1
>Reporter: Vince.Feng
>Priority: Minor
>
> When HDFS client with viewfs want to move file/directory to trash by
> RemoteUser or
> ProxyUser , it will fail. Codes like below:
> {code:java}
> // code placeholder
> Trash.moveToAppropriateTrash(fileSystem, path, conf);
> {code}
>
>
> 'fullyResolvedFs' will return a FileSystem with ugi of system user. But the
> source fileSystem ugi was specified account. So the moving to trash operation
> will fail. Because of trashRoot mismatch the real baseTrashPath.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15341) HDFS Move viewfs path to trash failed.
[
https://issues.apache.org/jira/browse/HDFS-15341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102540#comment-17102540
]
Vince.Feng commented on HDFS-15341:
---
Maybe the new method shoud be added to org.apache.hadoop.fs.Trash :
{code:java}
// code placeholder
public static boolean moveToAppropriateTrash(FileSystem fs, Path p,
Configuration conf,String user) throws IOException, InterruptedException {
Path fullyResolvedPath = fs.resolvePath(p);
// Get FileSystem by user.
FileSystem fullyResolvedFs =
FileSystem.get(fullyResolvedPath.toUri(), conf, user);
// If the trash interval is configured server side then clobber this
// configuration so that we always respect the server configuration.
try {
long trashInterval = fullyResolvedFs.getServerDefaults(
fullyResolvedPath).getTrashInterval();
if (0 != trashInterval) {
Configuration confCopy = new Configuration(conf);
confCopy.setLong(CommonConfigurationKeysPublic.FS_TRASH_INTERVAL_KEY,
trashInterval);
conf = confCopy;
}
} catch (Exception e) {
// If we can not determine that trash is enabled server side then
// bail rather than potentially deleting a file when trash is enabled.
LOG.warn("Failed to get server trash configuration", e);
throw new IOException("Failed to get server trash configuration", e);
}
Trash trash = new Trash(fullyResolvedFs, conf);
boolean success = trash.moveToTrash(fullyResolvedPath);
if (success) {
System.out.println("Moved: '" + p + "' to trash at: " +
trash.getCurrentTrashDir() );
}
return success;
}
{code}
> HDFS Move viewfs path to trash failed.
> --
>
> Key: HDFS-15341
> URL: https://issues.apache.org/jira/browse/HDFS-15341
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.10.0, 3.3.1
>Reporter: Vince.Feng
>Priority: Minor
>
> When HDFS client with viewfs want to move file/directory to trash by
> RemoteUser or
> ProxyUser , it will fail. Codes like below:
> {code:java}
> // code placeholder
> Trash.moveToAppropriateTrash(fileSystem, path, conf);
> {code}
>
>
> 'fullyResolvedFs' will return a FileSystem with ugi of system user. But the
> source fileSystem ugi was specified account. So the moving to trash operation
> will fail. Because of trashRoot mismatch the real baseTrashPath.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15341) HDFS Move viewfs path to trash failed.
[
https://issues.apache.org/jira/browse/HDFS-15341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vince.Feng updated HDFS-15341:
--
Description:
When HDFS client with viewfs want to move file/directory to trash by RemoteUser
or
ProxyUser , it will fail. Codes like below:
{code:java}
// code placeholder
Trash.moveToAppropriateTrash(fileSystem, path, conf);
{code}
'fullyResolvedFs' will return a FileSystem with ugi of system user. But the
source fileSystem ugi was specified account. So the moving to trash operation
will fail. Because of trashRoot mismatch the real baseTrashPath.
was:
When HDFS client with viewfs want to move file/directory to trash by RemoteUser
or
ProxyUser , it will fail. Codes like below:
{code:java}
// code placeholder
Trash.moveToAppropriateTrash(fileSystem, path, conf);
{code}
'fullyResolvedFs' will return a FileSystem with account of 'HADOOP_USER_NAME‘.
But the source fileSystem take the account of Specified account. So the moving
to trash operation will fail. Because of trashRoot mismatch the real
baseTrashPath.
> HDFS Move viewfs path to trash failed.
> --
>
> Key: HDFS-15341
> URL: https://issues.apache.org/jira/browse/HDFS-15341
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.10.0, 3.3.1
>Reporter: Vince.Feng
>Priority: Minor
>
> When HDFS client with viewfs want to move file/directory to trash by
> RemoteUser or
> ProxyUser , it will fail. Codes like below:
> {code:java}
> // code placeholder
> Trash.moveToAppropriateTrash(fileSystem, path, conf);
> {code}
>
>
> 'fullyResolvedFs' will return a FileSystem with ugi of system user. But the
> source fileSystem ugi was specified account. So the moving to trash operation
> will fail. Because of trashRoot mismatch the real baseTrashPath.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15338) listOpenFiles() should throw InvalidPathException to remind the admin.
[
https://issues.apache.org/jira/browse/HDFS-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102435#comment-17102435
]
Hadoop QA commented on HDFS-15338:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
6s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
44s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 10s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
181 unchanged - 0 fixed = 182 total (was 181) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 33s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 42s{color}
| {color:red} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}118m
50s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}228m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.ha.TestConfiguredFailoverProxyProvider |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-HDFS-Build/29253/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-15338 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13002370/HDFS-15338.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc
[jira] [Commented] (HDFS-15343) TestConfiguredFailoverProxyProvider is failing
[
https://issues.apache.org/jira/browse/HDFS-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102375#comment-17102375
]
Hudson commented on HDFS-15343:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18229 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18229/])
HDFS-15343. TestConfiguredFailoverProxyProvider is failing. (#2001) (github:
rev c784ba370eeb3e48c1c2c471ca72674c62a4edb7)
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/MockDomainNameResolver.java
> TestConfiguredFailoverProxyProvider is failing
> --
>
> Key: HDFS-15343
> URL: https://issues.apache.org/jira/browse/HDFS-15343
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Fix For: 3.3.0
>
>
> TestConfiguredFailoverProxyProvider.testResolveDomainNameUsingDNS is failing.
> {noformat}
> java.lang.AssertionError: nn2 wasn't returned:
> {host02.com/154.212.43.145:8020=31, host01.com:8020=19}
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestConfiguredFailoverProxyProvider.testResolveDomainNameUsingDNS(TestConfiguredFailoverProxyProvider.java:298)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestConfiguredFailoverProxyProvider.testResolveDomainNameUsingDNS(TestConfiguredFailoverProxyProvider.java:320)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15343) TestConfiguredFailoverProxyProvider is failing
[
https://issues.apache.org/jira/browse/HDFS-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-15343:
-
Fix Version/s: 3.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Merged the PR into trunk, branch-3.3, and branch-3.3.0.
> TestConfiguredFailoverProxyProvider is failing
> --
>
> Key: HDFS-15343
> URL: https://issues.apache.org/jira/browse/HDFS-15343
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Fix For: 3.3.0
>
>
> TestConfiguredFailoverProxyProvider.testResolveDomainNameUsingDNS is failing.
> {noformat}
> java.lang.AssertionError: nn2 wasn't returned:
> {host02.com/154.212.43.145:8020=31, host01.com:8020=19}
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestConfiguredFailoverProxyProvider.testResolveDomainNameUsingDNS(TestConfiguredFailoverProxyProvider.java:298)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestConfiguredFailoverProxyProvider.testResolveDomainNameUsingDNS(TestConfiguredFailoverProxyProvider.java:320)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15294) RBF: Balance data across federation namespaces with DistCp and snapshot diff
[
https://issues.apache.org/jira/browse/HDFS-15294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jinglun updated HDFS-15294:
---
Attachment: distcp-balance.v2.pdf
> RBF: Balance data across federation namespaces with DistCp and snapshot diff
>
>
> Key: HDFS-15294
> URL: https://issues.apache.org/jira/browse/HDFS-15294
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: BalanceProcedureScheduler.png, HDFS-15294.001.patch,
> HDFS-15294.002.patch, HDFS-15294.003.patch, HDFS-15294.003.reupload.patch,
> HDFS-15294.004.patch, HDFS-15294.005.patch, HDFS-15294.006.patch,
> HDFS-15294.007.patch, distcp-balance.pdf, distcp-balance.v2.pdf
>
>
> This jira introduces a new balance command 'fedbalance' that is ran by the
> administrator. The process is:
> 1. Use distcp and snapshot diff to sync data between src and dst until they
> are the same.
> 2. Update mount table in Router.
> 3. Delete the src to trash.
>
> The patch is too big to review, so I split it into 2 patches:
> Phase 1 / The State Machine(BalanceProcedureScheduler): Including the
> abstraction of job and scheduler model.
> {code:java}
> org.apache.hadoop.hdfs.procedure.BalanceProcedureScheduler;
> org.apache.hadoop.hdfs.procedure.BalanceProcedureConfigKeys;
> org.apache.hadoop.hdfs.procedure.BalanceProcedure;
> org.apache.hadoop.hdfs.procedure.BalanceJob;
> org.apache.hadoop.hdfs.procedure.BalanceJournal;
> org.apache.hadoop.hdfs.procedure.HDFSJournal;
> {code}
> Phase 2 / The DistCpFedBalance: It's an implementation of BalanceJob.
>
> {code:java}
> org.apache.hadoop.hdfs.server.federation.procedure.MountTableProcedure;
> org.apache.hadoop.tools.DistCpFedBalance;
> org.apache.hadoop.tools.DistCpProcedure;
> org.apache.hadoop.tools.FedBalance;
> org.apache.hadoop.tools.FedBalanceConfigs;
> org.apache.hadoop.tools.FedBalanceContext;
> org.apache.hadoop.tools.TrashProcedure;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
