[
https://issues.apache.org/jira/browse/HDFS-16477?focusedWorklogId=748308&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-748308
]
ASF GitHub Bot logged work on HDFS-16477:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 27/Mar/22 07:36
Start Date: 27/Mar/22 07:36
Worklog Time Spent: 10m
Work Description: tomscut commented on a change in pull request #4009:
URL: https://github.com/apache/hadoop/pull/4009#discussion_r835865303
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/sps/TestExternalStoragePolicySatisfier.java
##########
@@ -441,6 +442,8 @@ private void doTestWhenStoragePolicySetToCOLD() throws
Exception {
hdfsCluster.triggerHeartbeats();
dfs.satisfyStoragePolicy(new Path(FILE));
+ // Assert metrics.
+ assertEquals(1, hdfsCluster.getNamesystem().getPendingSPSPaths());
// Wait till namenode notified about the block location details
DFSTestUtil.waitExpectedStorageType(FILE, StorageType.ARCHIVE, 3, 35000,
dfs);
Review comment:
> ```
> // Wait till namenode notified about the block location details
> DFSTestUtil.waitExpectedStorageType(FILE, StorageType.ARCHIVE, 3,
35000,
> dfs);
> ```
>
> Here you are waiting for SPS to process the path and move the blocks to
the correct place, once this is done, whether `getPendingSPSPaths` will still
return 1? I suppose no, right? the path got processed so the count should
reduce to 0.
>
> So, my take is you don't have a control on `
DFSTestUtil.waitExpectedStorageType(FILE, StorageType.ARCHIVE, 3, 35000,
dfs);`, if by chance SPS process that path before your assertion then the test
will fail.
>
> I haven't gone through the code, but that is what I felt in my initial
pass, if it doesn't work this way do lemme know
Thank you @ayushtkn for your detailed explanation. I made a mistake and you
are right. I'll add a new unit test to assert metrics without running SPS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 748308)
Time Spent: 4h 10m (was: 4h)
> [SPS]: Add metric PendingSPSPaths for getting the number of paths to be
> processed by SPS
> ----------------------------------------------------------------------------------------
>
> Key: HDFS-16477
> URL: https://issues.apache.org/jira/browse/HDFS-16477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: tomscut
> Assignee: tomscut
> Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Currently we have no idea how many paths are waiting to be processed when
> using the SPS feature. We should add metric PendingSPSPaths for getting the
> number of paths to be processed by SPS in NameNode.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
