[jira] [Work logged] (HDFS-16076) Avoid using slow DataNodes for reading by sorting locations
[ https://issues.apache.org/jira/browse/HDFS-16076?focusedWorklogId=614312=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-614312 ] ASF GitHub Bot logged work on HDFS-16076: - Author: ASF GitHub Bot Created on: 24/Jun/21 02:30 Start Date: 24/Jun/21 02:30 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3117: URL: https://github.com/apache/hadoop/pull/3117#issuecomment-867285100 > Merged it. Thanks for your contribution, @tomscut. Thanks @tasanuma . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 614312) Time Spent: 2h 20m (was: 2h 10m) > Avoid using slow DataNodes for reading by sorting locations > --- > > Key: HDFS-16076 > URL: https://issues.apache.org/jira/browse/HDFS-16076 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > After sorting the expected location list will be: live -> slow -> stale -> > staleAndSlow -> entering_maintenance -> decommissioned. This reduces the > probability that slow nodes will be used for reading. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16076) Avoid using slow DataNodes for reading by sorting locations
[ https://issues.apache.org/jira/browse/HDFS-16076?focusedWorklogId=614311=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-614311 ] ASF GitHub Bot logged work on HDFS-16076: - Author: ASF GitHub Bot Created on: 24/Jun/21 02:28 Start Date: 24/Jun/21 02:28 Worklog Time Spent: 10m Work Description: tasanuma commented on pull request #3117: URL: https://github.com/apache/hadoop/pull/3117#issuecomment-867284477 Merged it. Thanks for your contribution, @tomscut. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 614311) Time Spent: 2h 10m (was: 2h) > Avoid using slow DataNodes for reading by sorting locations > --- > > Key: HDFS-16076 > URL: https://issues.apache.org/jira/browse/HDFS-16076 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > After sorting the expected location list will be: live -> slow -> stale -> > staleAndSlow -> entering_maintenance -> decommissioned. This reduces the > probability that slow nodes will be used for reading. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16076) Avoid using slow DataNodes for reading by sorting locations
[ https://issues.apache.org/jira/browse/HDFS-16076?focusedWorklogId=614310=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-614310 ] ASF GitHub Bot logged work on HDFS-16076: - Author: ASF GitHub Bot Created on: 24/Jun/21 02:27 Start Date: 24/Jun/21 02:27 Worklog Time Spent: 10m Work Description: tasanuma merged pull request #3117: URL: https://github.com/apache/hadoop/pull/3117 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 614310) Time Spent: 2h (was: 1h 50m) > Avoid using slow DataNodes for reading by sorting locations > --- > > Key: HDFS-16076 > URL: https://issues.apache.org/jira/browse/HDFS-16076 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > After sorting the expected location list will be: live -> slow -> stale -> > staleAndSlow -> entering_maintenance -> decommissioned. This reduces the > probability that slow nodes will be used for reading. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16076) Avoid using slow DataNodes for reading by sorting locations
[ https://issues.apache.org/jira/browse/HDFS-16076?focusedWorklogId=613913=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-613913 ] ASF GitHub Bot logged work on HDFS-16076: - Author: ASF GitHub Bot Created on: 23/Jun/21 09:29 Start Date: 23/Jun/21 09:29 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3117: URL: https://github.com/apache/hadoop/pull/3117#issuecomment-866681953 Thanks @tasanuma for your second review. These failed UTs work fine locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 613913) Time Spent: 1h 50m (was: 1h 40m) > Avoid using slow DataNodes for reading by sorting locations > --- > > Key: HDFS-16076 > URL: https://issues.apache.org/jira/browse/HDFS-16076 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > After sorting the expected location list will be: live -> slow -> stale -> > staleAndSlow -> entering_maintenance -> decommissioned. This reduces the > probability that slow nodes will be used for reading. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16076) Avoid using slow DataNodes for reading by sorting locations
[
https://issues.apache.org/jira/browse/HDFS-16076?focusedWorklogId=613895=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-613895
]
ASF GitHub Bot logged work on HDFS-16076:
-
Author: ASF GitHub Bot
Created on: 23/Jun/21 08:43
Start Date: 23/Jun/21 08:43
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3117:
URL: https://github.com/apache/hadoop/pull/3117#issuecomment-866649231
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 7s | | trunk passed |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 24s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 53s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 346m 40s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3117/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 4s | | The patch does not
generate ASF License warnings. |
| | | 438m 51s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestDecommissioningStatus
|
| | hadoop.hdfs.TestDFSShell |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3117/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3117 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 28e07ef0078f 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 74bb4aa36cc2f934905878f70762e628de073e93 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
[jira] [Work logged] (HDFS-16076) Avoid using slow DataNodes for reading by sorting locations
[
https://issues.apache.org/jira/browse/HDFS-16076?focusedWorklogId=613773=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-613773
]
ASF GitHub Bot logged work on HDFS-16076:
-
Author: ASF GitHub Bot
Created on: 23/Jun/21 01:14
Start Date: 23/Jun/21 01:14
Worklog Time Spent: 10m
Work Description: tomscut commented on a change in pull request #3117:
URL: https://github.com/apache/hadoop/pull/3117#discussion_r656688215
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestSortLocatedBlock.java
##
@@ -126,11 +119,189 @@ public void testWithMultipleStateDatanodes() {
&& decommissionedNodes.contains(locations[4]));
}
- private static DatanodeManager mockDatanodeManager() throws IOException {
+ /**
+ * Test to verify sorting with multiple state
+ * datanodes exists in storage lists.
+ *
+ * After sorting the expected datanodes list will be:
+ * live -> slow -> stale -> staleAndSlow ->
+ * entering_maintenance -> decommissioned.
+ *
+ * avoidStaleDataNodesForRead=true && avoidSlowDataNodesForRead=true
+ * (d5 -> d4 -> d3 -> d2 -> d1 -> d0)
+ */
+ @Test(timeout = 3)
+ public void testAviodStaleAndSlowDatanodes() throws IOException {
+DatanodeManager dm = mockDatanodeManager(true, true);
+DatanodeInfo[] locs = mockDatanodes(dm);
+
+ArrayList locatedBlocks = new ArrayList<>();
+locatedBlocks.add(new LocatedBlock(
+new ExtendedBlock("pool", Long.MIN_VALUE,
+1024L, new Date().getTime()), locs));
+
+// sort located blocks
+dm.sortLocatedBlocks(null, locatedBlocks);
+
+// get locations after sorting
+LocatedBlock locatedBlock = locatedBlocks.get(0);
+DatanodeInfoWithStorage[] locations = locatedBlock.getLocations();
+
+// assert location order:
+// live -> stale -> entering_maintenance -> decommissioned
+// live
+assertEquals(locs[5].getIpAddr(), locations[0].getIpAddr());
+// slow
+assertEquals(locs[4].getIpAddr(), locations[1].getIpAddr());
+// stale
+assertEquals(locs[3].getIpAddr(), locations[2].getIpAddr());
+// stale and slow
+assertEquals(locs[2].getIpAddr(), locations[3].getIpAddr());
+// entering_maintenance
+assertEquals(locs[1].getIpAddr(), locations[4].getIpAddr());
+// decommissioned
+assertEquals(locs[0].getIpAddr(), locations[5].getIpAddr());
+ }
+
+ /**
+ * Test to verify sorting with multiple state
+ * datanodes exists in storage lists.
+ *
+ * After sorting the expected datanodes list will be:
+ * (live <-> slow) -> (stale <-> staleAndSlow) ->
+ * entering_maintenance -> decommissioned.
+ *
+ * avoidStaleDataNodesForRead=true && avoidSlowDataNodesForRead=false
+ * (d5 <-> d4) -> (d3 <-> d2) -> d1 -> d0
+ */
+ @Test(timeout = 3)
+ public void testAviodStaleDatanodes() throws IOException {
+DatanodeManager dm = mockDatanodeManager(true, false);
+DatanodeInfo[] locs = mockDatanodes(dm);
+
+ArrayList locatedBlocks = new ArrayList<>();
+locatedBlocks.add(new LocatedBlock(
+new ExtendedBlock("pool", Long.MIN_VALUE,
+1024L, new Date().getTime()), locs));
+
+// sort located blocks
+dm.sortLocatedBlocks(null, locatedBlocks);
+
+// get locations after sorting
+LocatedBlock locatedBlock = locatedBlocks.get(0);
+DatanodeInfoWithStorage[] locations = locatedBlock.getLocations();
+
+// assert location order:
+// live -> stale -> entering_maintenance -> decommissioned
+// live
+assertTrue((locs[5].getIpAddr() == locations[0].getIpAddr() &&
+locs[4].getIpAddr() == locations[1].getIpAddr()) ||
+(locs[5].getIpAddr() == locations[1].getIpAddr() &&
+locs[4].getIpAddr() == locations[0].getIpAddr()));
+// stale
+assertTrue((locs[3].getIpAddr() == locations[2].getIpAddr() &&
+locs[2].getIpAddr() == locations[3].getIpAddr()) ||
+(locs[3].getIpAddr() == locations[3].getIpAddr() &&
+locs[2].getIpAddr() == locations[2].getIpAddr()));
+// entering_maintenance
+assertEquals(locs[1].getIpAddr(), locations[4].getIpAddr());
+// decommissioned
+assertEquals(locs[0].getIpAddr(), locations[5].getIpAddr());
+ }
+
+ /**
+ * Test to verify sorting with multiple state
+ * datanodes exists in storage lists.
+ *
+ * After sorting the expected datanodes list will be:
+ * (live <-> stale) -> (slow <-> staleAndSlow) ->
+ * entering_maintenance -> decommissioned.
+ *
+ * avoidStaleDataNodesForRead=false && avoidSlowDataNodesForRead=true
+ * (d5 -> d3) -> ( d4 <-> d2) -> d1 -> d0)
+ */
+ @Test(timeout = 3)
+ public void testAviodSlowDatanodes() throws IOException {
+DatanodeManager dm = mockDatanodeManager(false, true);
+DatanodeInfo[] locs = mockDatanodes(dm);
+
+ArrayList
[jira] [Work logged] (HDFS-16076) Avoid using slow DataNodes for reading by sorting locations
[
https://issues.apache.org/jira/browse/HDFS-16076?focusedWorklogId=613615=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-613615
]
ASF GitHub Bot logged work on HDFS-16076:
-
Author: ASF GitHub Bot
Created on: 22/Jun/21 17:34
Start Date: 22/Jun/21 17:34
Worklog Time Spent: 10m
Work Description: tasanuma commented on a change in pull request #3117:
URL: https://github.com/apache/hadoop/pull/3117#discussion_r656413524
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestSortLocatedBlock.java
##
@@ -126,11 +119,189 @@ public void testWithMultipleStateDatanodes() {
&& decommissionedNodes.contains(locations[4]));
}
- private static DatanodeManager mockDatanodeManager() throws IOException {
+ /**
+ * Test to verify sorting with multiple state
+ * datanodes exists in storage lists.
+ *
+ * After sorting the expected datanodes list will be:
+ * live -> slow -> stale -> staleAndSlow ->
+ * entering_maintenance -> decommissioned.
+ *
+ * avoidStaleDataNodesForRead=true && avoidSlowDataNodesForRead=true
+ * (d5 -> d4 -> d3 -> d2 -> d1 -> d0)
+ */
+ @Test(timeout = 3)
+ public void testAviodStaleAndSlowDatanodes() throws IOException {
+DatanodeManager dm = mockDatanodeManager(true, true);
+DatanodeInfo[] locs = mockDatanodes(dm);
+
+ArrayList locatedBlocks = new ArrayList<>();
+locatedBlocks.add(new LocatedBlock(
+new ExtendedBlock("pool", Long.MIN_VALUE,
+1024L, new Date().getTime()), locs));
+
+// sort located blocks
+dm.sortLocatedBlocks(null, locatedBlocks);
+
+// get locations after sorting
+LocatedBlock locatedBlock = locatedBlocks.get(0);
+DatanodeInfoWithStorage[] locations = locatedBlock.getLocations();
+
+// assert location order:
+// live -> stale -> entering_maintenance -> decommissioned
+// live
+assertEquals(locs[5].getIpAddr(), locations[0].getIpAddr());
+// slow
+assertEquals(locs[4].getIpAddr(), locations[1].getIpAddr());
+// stale
+assertEquals(locs[3].getIpAddr(), locations[2].getIpAddr());
+// stale and slow
+assertEquals(locs[2].getIpAddr(), locations[3].getIpAddr());
+// entering_maintenance
+assertEquals(locs[1].getIpAddr(), locations[4].getIpAddr());
+// decommissioned
+assertEquals(locs[0].getIpAddr(), locations[5].getIpAddr());
+ }
+
+ /**
+ * Test to verify sorting with multiple state
+ * datanodes exists in storage lists.
+ *
+ * After sorting the expected datanodes list will be:
+ * (live <-> slow) -> (stale <-> staleAndSlow) ->
+ * entering_maintenance -> decommissioned.
+ *
+ * avoidStaleDataNodesForRead=true && avoidSlowDataNodesForRead=false
+ * (d5 <-> d4) -> (d3 <-> d2) -> d1 -> d0
+ */
+ @Test(timeout = 3)
+ public void testAviodStaleDatanodes() throws IOException {
+DatanodeManager dm = mockDatanodeManager(true, false);
+DatanodeInfo[] locs = mockDatanodes(dm);
+
+ArrayList locatedBlocks = new ArrayList<>();
+locatedBlocks.add(new LocatedBlock(
+new ExtendedBlock("pool", Long.MIN_VALUE,
+1024L, new Date().getTime()), locs));
+
+// sort located blocks
+dm.sortLocatedBlocks(null, locatedBlocks);
+
+// get locations after sorting
+LocatedBlock locatedBlock = locatedBlocks.get(0);
+DatanodeInfoWithStorage[] locations = locatedBlock.getLocations();
+
+// assert location order:
+// live -> stale -> entering_maintenance -> decommissioned
+// live
+assertTrue((locs[5].getIpAddr() == locations[0].getIpAddr() &&
+locs[4].getIpAddr() == locations[1].getIpAddr()) ||
+(locs[5].getIpAddr() == locations[1].getIpAddr() &&
+locs[4].getIpAddr() == locations[0].getIpAddr()));
+// stale
+assertTrue((locs[3].getIpAddr() == locations[2].getIpAddr() &&
+locs[2].getIpAddr() == locations[3].getIpAddr()) ||
+(locs[3].getIpAddr() == locations[3].getIpAddr() &&
+locs[2].getIpAddr() == locations[2].getIpAddr()));
+// entering_maintenance
+assertEquals(locs[1].getIpAddr(), locations[4].getIpAddr());
+// decommissioned
+assertEquals(locs[0].getIpAddr(), locations[5].getIpAddr());
+ }
+
+ /**
+ * Test to verify sorting with multiple state
+ * datanodes exists in storage lists.
+ *
+ * After sorting the expected datanodes list will be:
+ * (live <-> stale) -> (slow <-> staleAndSlow) ->
+ * entering_maintenance -> decommissioned.
+ *
+ * avoidStaleDataNodesForRead=false && avoidSlowDataNodesForRead=true
+ * (d5 -> d3) -> ( d4 <-> d2) -> d1 -> d0)
Review comment:
```suggestion
* (d5 -> d3) -> (d4 <-> d2) -> d1 -> d0
```
##
File path:
[jira] [Work logged] (HDFS-16076) Avoid using slow DataNodes for reading by sorting locations
[ https://issues.apache.org/jira/browse/HDFS-16076?focusedWorklogId=613026=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-613026 ] ASF GitHub Bot logged work on HDFS-16076: - Author: ASF GitHub Bot Created on: 22/Jun/21 07:38 Start Date: 22/Jun/21 07:38 Worklog Time Spent: 10m Work Description: tomscut edited a comment on pull request #3117: URL: https://github.com/apache/hadoop/pull/3117#issuecomment-864670631 Hi @tasanuma @jojochuang @goiri @Hexiaoqiao @ayushtkn , could you please help to review the code? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 613026) Time Spent: 1h 10m (was: 1h) > Avoid using slow DataNodes for reading by sorting locations > --- > > Key: HDFS-16076 > URL: https://issues.apache.org/jira/browse/HDFS-16076 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > After sorting the expected location list will be: live -> slow -> stale -> > staleAndSlow -> entering_maintenance -> decommissioned. This reduces the > probability that slow nodes will be used for reading. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16076) Avoid using slow DataNodes for reading by sorting locations
[ https://issues.apache.org/jira/browse/HDFS-16076?focusedWorklogId=612515=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-612515 ] ASF GitHub Bot logged work on HDFS-16076: - Author: ASF GitHub Bot Created on: 21/Jun/21 09:34 Start Date: 21/Jun/21 09:34 Worklog Time Spent: 10m Work Description: tomscut edited a comment on pull request #3117: URL: https://github.com/apache/hadoop/pull/3117#issuecomment-864670631 Hi @tasanuma @jojochuang @goiri @Hexiaoqiao @ayushtkn , could you please help to review the code? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 612515) Time Spent: 1h (was: 50m) > Avoid using slow DataNodes for reading by sorting locations > --- > > Key: HDFS-16076 > URL: https://issues.apache.org/jira/browse/HDFS-16076 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > After sorting the expected location list will be: live -> slow -> stale -> > staleAndSlow -> entering_maintenance -> decommissioned. This reduces the > probability that slow nodes will be used for reading. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16076) Avoid using slow DataNodes for reading by sorting locations
[ https://issues.apache.org/jira/browse/HDFS-16076?focusedWorklogId=612411=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-612411 ] ASF GitHub Bot logged work on HDFS-16076: - Author: ASF GitHub Bot Created on: 21/Jun/21 02:05 Start Date: 21/Jun/21 02:05 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3117: URL: https://github.com/apache/hadoop/pull/3117#issuecomment-864670631 Hi @tasanuma @jojochuang @goiri @Hexiaoqiao @ayushtkn , could you please to review the code? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 612411) Time Spent: 50m (was: 40m) > Avoid using slow DataNodes for reading by sorting locations > --- > > Key: HDFS-16076 > URL: https://issues.apache.org/jira/browse/HDFS-16076 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > After sorting the expected location list will be: live -> slow -> stale -> > staleAndSlow -> entering_maintenance -> decommissioned. This reduces the > probability that slow nodes will be used for reading. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16076) Avoid using slow DataNodes for reading by sorting locations
[ https://issues.apache.org/jira/browse/HDFS-16076?focusedWorklogId=612410=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-612410 ] ASF GitHub Bot logged work on HDFS-16076: - Author: ASF GitHub Bot Created on: 21/Jun/21 02:02 Start Date: 21/Jun/21 02:02 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3117: URL: https://github.com/apache/hadoop/pull/3117#issuecomment-864669265 Those failed unit tests work fine locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 612410) Time Spent: 40m (was: 0.5h) > Avoid using slow DataNodes for reading by sorting locations > --- > > Key: HDFS-16076 > URL: https://issues.apache.org/jira/browse/HDFS-16076 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > After sorting the expected location list will be: live -> slow -> stale -> > staleAndSlow -> entering_maintenance -> decommissioned. This reduces the > probability that slow nodes will be used for reading. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16076) Avoid using slow DataNodes for reading by sorting locations
[
https://issues.apache.org/jira/browse/HDFS-16076?focusedWorklogId=612290=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-612290
]
ASF GitHub Bot logged work on HDFS-16076:
-
Author: ASF GitHub Bot
Created on: 19/Jun/21 12:05
Start Date: 19/Jun/21 12:05
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3117:
URL: https://github.com/apache/hadoop/pull/3117#issuecomment-864397544
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 13s | | trunk passed |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 58s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 11s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 11s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 17s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 384m 59s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3117/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 477m 36s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestDecommissioningStatus
|
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
| | hadoop.hdfs.TestViewDistributedFileSystem |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks |
| | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer |
| | hadoop.hdfs.TestInjectionForSimulatedStorage |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.TestWriteReadStripedFile |
| | hadoop.hdfs.tools.TestDebugAdmin |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3117/4/artifact/out/Dockerfile
|
| GITHUB PR |
[jira] [Work logged] (HDFS-16076) Avoid using slow DataNodes for reading by sorting locations
[ https://issues.apache.org/jira/browse/HDFS-16076?focusedWorklogId=612252=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-612252 ] ASF GitHub Bot logged work on HDFS-16076: - Author: ASF GitHub Bot Created on: 19/Jun/21 04:09 Start Date: 19/Jun/21 04:09 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3117: URL: https://github.com/apache/hadoop/pull/3117#issuecomment-864352035 Rebased to the latest commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 612252) Time Spent: 20m (was: 10m) > Avoid using slow DataNodes for reading by sorting locations > --- > > Key: HDFS-16076 > URL: https://issues.apache.org/jira/browse/HDFS-16076 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > After sorting the expected location list will be: live -> slow -> stale -> > staleAndSlow -> entering_maintenance -> decommissioned. This reduces the > probability that slow nodes will be used for reading. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16076) Avoid using slow DataNodes for reading by sorting locations
[ https://issues.apache.org/jira/browse/HDFS-16076?focusedWorklogId=611890=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-611890 ] ASF GitHub Bot logged work on HDFS-16076: - Author: ASF GitHub Bot Created on: 18/Jun/21 20:53 Start Date: 18/Jun/21 20:53 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #3117: URL: https://github.com/apache/hadoop/pull/3117#issuecomment-863612534 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 611890) Remaining Estimate: 0h Time Spent: 10m > Avoid using slow DataNodes for reading by sorting locations > --- > > Key: HDFS-16076 > URL: https://issues.apache.org/jira/browse/HDFS-16076 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > After sorting the expected location list will be: live -> slow -> stale -> > staleAndSlow -> entering_maintenance -> decommissioned. This reduces the > probability that slow nodes will be used for reading. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
