[jira] [Updated] (HDFS-15739) Missing Javadoc for a param in DFSNetworkTopology
[ https://issues.apache.org/jira/browse/HDFS-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhanghuazong updated HDFS-15739: Attachment: (was: HDFS-15739.0.patch) > Missing Javadoc for a param in DFSNetworkTopology > - > > Key: HDFS-15739 > URL: https://issues.apache.org/jira/browse/HDFS-15739 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: zhanghuazong >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-15739.0.patch > > Time Spent: 50m > Remaining Estimate: 0h > > Only add missing Javadoc for a param in method chooseRandomWithStorageType > of DFSNetworkTopology.java. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15739) Missing Javadoc for a param in DFSNetworkTopology
[ https://issues.apache.org/jira/browse/HDFS-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhanghuazong updated HDFS-15739: Attachment: HDFS-15739.0.patch > Missing Javadoc for a param in DFSNetworkTopology > - > > Key: HDFS-15739 > URL: https://issues.apache.org/jira/browse/HDFS-15739 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: zhanghuazong >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-15739.0.patch > > Time Spent: 50m > Remaining Estimate: 0h > > Only add missing Javadoc for a param in method chooseRandomWithStorageType > of DFSNetworkTopology.java. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15739) Missing Javadoc for a param in DFSNetworkTopology
[ https://issues.apache.org/jira/browse/HDFS-15739?focusedWorklogId=526224=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-526224 ] ASF GitHub Bot logged work on HDFS-15739: - Author: ASF GitHub Bot Created on: 19/Dec/20 06:10 Start Date: 19/Dec/20 06:10 Worklog Time Spent: 10m Work Description: langlaile1221 opened a new pull request #2566: URL: https://github.com/apache/hadoop/pull/2566 Only add missing Javadoc for a param in method chooseRandomWithStorageType of DFSNetworkTopology.java. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 526224) Time Spent: 50m (was: 40m) > Missing Javadoc for a param in DFSNetworkTopology > - > > Key: HDFS-15739 > URL: https://issues.apache.org/jira/browse/HDFS-15739 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: zhanghuazong >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-15739.0.patch > > Time Spent: 50m > Remaining Estimate: 0h > > Only add missing Javadoc for a param in method chooseRandomWithStorageType > of DFSNetworkTopology.java. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15739) Missing Javadoc for a param in DFSNetworkTopology
[ https://issues.apache.org/jira/browse/HDFS-15739?focusedWorklogId=526223=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-526223 ] ASF GitHub Bot logged work on HDFS-15739: - Author: ASF GitHub Bot Created on: 19/Dec/20 06:02 Start Date: 19/Dec/20 06:02 Worklog Time Spent: 10m Work Description: langlaile1221 closed pull request #2565: URL: https://github.com/apache/hadoop/pull/2565 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 526223) Time Spent: 40m (was: 0.5h) > Missing Javadoc for a param in DFSNetworkTopology > - > > Key: HDFS-15739 > URL: https://issues.apache.org/jira/browse/HDFS-15739 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: zhanghuazong >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-15739.0.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Only add missing Javadoc for a param in method chooseRandomWithStorageType > of DFSNetworkTopology.java. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15739) Missing Javadoc for a param in DFSNetworkTopology
[ https://issues.apache.org/jira/browse/HDFS-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhanghuazong updated HDFS-15739: Description: Only add missing Javadoc for a param in method chooseRandomWithStorageType of DFSNetworkTopology.java. was: Only add missing Javadoc for a param in method chooseRandomWithStorageType of AppSchedulingInfo.java. > Missing Javadoc for a param in DFSNetworkTopology > - > > Key: HDFS-15739 > URL: https://issues.apache.org/jira/browse/HDFS-15739 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: zhanghuazong >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-15739.0.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Only add missing Javadoc for a param in method chooseRandomWithStorageType > of DFSNetworkTopology.java. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15739) Missing Javadoc for a param in DFSNetworkTopology
[ https://issues.apache.org/jira/browse/HDFS-15739?focusedWorklogId=526211=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-526211 ] ASF GitHub Bot logged work on HDFS-15739: - Author: ASF GitHub Bot Created on: 19/Dec/20 03:41 Start Date: 19/Dec/20 03:41 Worklog Time Spent: 10m Work Description: runitao commented on pull request #2565: URL: https://github.com/apache/hadoop/pull/2565#issuecomment-748413464 +1 LGTM. The failed UTs are unrelated with this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 526211) Time Spent: 0.5h (was: 20m) > Missing Javadoc for a param in DFSNetworkTopology > - > > Key: HDFS-15739 > URL: https://issues.apache.org/jira/browse/HDFS-15739 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: zhanghuazong >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-15739.0.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Only add missing Javadoc for a param in method chooseRandomWithStorageType > of AppSchedulingInfo.java. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15737) Don't remove datanodes from outOfServiceNodeBlocks while checking in DatanodeAdminManager
[
https://issues.apache.org/jira/browse/HDFS-15737?focusedWorklogId=526158=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-526158
]
ASF GitHub Bot logged work on HDFS-15737:
-
Author: ASF GitHub Bot
Created on: 18/Dec/20 21:54
Start Date: 18/Dec/20 21:54
Worklog Time Spent: 10m
Work Description: NickyYe commented on pull request #2562:
URL: https://github.com/apache/hadoop/pull/2562#issuecomment-748338149
> Thanks for the information - this may explain why HDFS-12703 was needed,
as some exceptions which were not logged at that time, caused the decommission
thread to stop running until the NN was restarted. The change there was to
catch the exception.
>
> The change here looks correct to me, but as the issue exists on the trunk
branch, we should fix it there first, and then backport to 3.3, 3.2, 3.1 and
2.10 so the fix is in place across all branches.
Due to HDFS-14854, the fix on trunk could be a very different one, since it
doesn't make sense to change the new interface with a boolean parameter to
stopTrackingNode while DatanodeAdminBackoffMonitor does't need.
Looks a better fix would be introduce a cancelledNodes to
DatanodeAdminDefaultMonitor, just like DatanodeAdminBackoffMonitor . Then in
stopTrackingNode, don't remove dn from outOfServiceNodeBlocks, but add it to
cancelledNodes for further process.
However, the change would be a little bit bigger.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 526158)
Time Spent: 1.5h (was: 1h 20m)
> Don't remove datanodes from outOfServiceNodeBlocks while checking in
> DatanodeAdminManager
> -
>
> Key: HDFS-15737
> URL: https://issues.apache.org/jira/browse/HDFS-15737
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Ye Ni
>Assignee: Ye Ni
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.10.2
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> With CyclicIteration, remove an item while iterating causes either dead loop
> or ConcurrentModificationException.
> This item should be removed by
> {{toRemove.add(dn);}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15737) Don't remove datanodes from outOfServiceNodeBlocks while checking in DatanodeAdminManager
[
https://issues.apache.org/jira/browse/HDFS-15737?focusedWorklogId=526143=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-526143
]
ASF GitHub Bot logged work on HDFS-15737:
-
Author: ASF GitHub Bot
Created on: 18/Dec/20 21:04
Start Date: 18/Dec/20 21:04
Worklog Time Spent: 10m
Work Description: sodonnel commented on pull request #2562:
URL: https://github.com/apache/hadoop/pull/2562#issuecomment-748318475
Thanks for the information - this may explain why HDFS-12703 was needed, as
some exceptions which were not logged at that time, caused the decommission
thread to stop running until the NN was restarted. The change there was to
catch the exception.
The change here looks correct to me, but as the issue exists on the trunk
branch, we should fix it there first, and then backport to 3.3, 3.2, 3.1 and
2.10 so the fix is in place across all branches.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 526143)
Time Spent: 1h 20m (was: 1h 10m)
> Don't remove datanodes from outOfServiceNodeBlocks while checking in
> DatanodeAdminManager
> -
>
> Key: HDFS-15737
> URL: https://issues.apache.org/jira/browse/HDFS-15737
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Ye Ni
>Assignee: Ye Ni
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.10.2
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> With CyclicIteration, remove an item while iterating causes either dead loop
> or ConcurrentModificationException.
> This item should be removed by
> {{toRemove.add(dn);}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15737) Don't remove datanodes from outOfServiceNodeBlocks while checking in DatanodeAdminManager
[
https://issues.apache.org/jira/browse/HDFS-15737?focusedWorklogId=526123=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-526123
]
ASF GitHub Bot logged work on HDFS-15737:
-
Author: ASF GitHub Bot
Created on: 18/Dec/20 19:43
Start Date: 18/Dec/20 19:43
Worklog Time Spent: 10m
Work Description: NickyYe commented on pull request #2562:
URL: https://github.com/apache/hadoop/pull/2562#issuecomment-748285060
> ConcurrentModificationException
Thanks for looking. If there are only 2 datanodes in outOfServiceNodeBlocks
and the first one is removed, then it will be a dead loop on the second
datanode. If there are more than 2 datanodes and the first one is removed,
there will be a ConcurrentModificationException. I see both two cases in our
prod very often.
This issue only happens when remove (dnAdmin.stopMaintenance(dn);). By
outOfServiceNodeBlocks.put(dn, blocks), it only updates the value, so Cyclic
Iteration won't be affected
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 526123)
Time Spent: 1h 10m (was: 1h)
> Don't remove datanodes from outOfServiceNodeBlocks while checking in
> DatanodeAdminManager
> -
>
> Key: HDFS-15737
> URL: https://issues.apache.org/jira/browse/HDFS-15737
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Ye Ni
>Assignee: Ye Ni
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.10.2
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> With CyclicIteration, remove an item while iterating causes either dead loop
> or ConcurrentModificationException.
> This item should be removed by
> {{toRemove.add(dn);}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15308) TestReconstructStripedFile#testNNSendsErasureCodingTasks fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251957#comment-17251957
]
Ahmed Hussein commented on HDFS-15308:
--
We have {{dfs.namenode.redundancy.considerLoad}} set to false in HDFS-15659
I guess the failure needs to be re-evaluated before merging this patch.
> TestReconstructStripedFile#testNNSendsErasureCodingTasks fails intermittently
> -
>
> Key: HDFS-15308
> URL: https://issues.apache.org/jira/browse/HDFS-15308
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.3.0
>Reporter: Toshihiko Uchida
>Assignee: Hemanth Boyina
>Priority: Blocker
> Labels: flaky-test
> Attachments: HDFS-15308.001.patch, HDFS-15308.002.patch
>
>
> In HDFS-14353, TestReconstructStripedFile.testNNSendsErasureCodingTasks
> failed once due to pending reconstruction timeout as follows.
> {code}
> java.lang.AssertionError: Found 4 timeout pending reconstruction tasks
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hdfs.TestReconstructStripedFile.testNNSendsErasureCodingTasks(TestReconstructStripedFile.java:502)
> at
> org.apache.hadoop.hdfs.TestReconstructStripedFile.testNNSendsErasureCodingTasks(TestReconstructStripedFile.java:458)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> The error occurred on the following assertion.
> {code}
> // Make sure that all pending reconstruction tasks can be processed.
> while (ns.getPendingReconstructionBlocks() > 0) {
> long timeoutPending = ns.getNumTimedOutPendingReconstructions();
> assertTrue(String.format("Found %d timeout pending reconstruction tasks",
> timeoutPending), timeoutPending == 0);
> Thread.sleep(1000);
> }
> {code}
> The failure could not be reproduced in the reporter's docker environment
> (start-build-environment.sh).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15679) DFSOutputStream close should be a no-op when called multiple times
[
https://issues.apache.org/jira/browse/HDFS-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251942#comment-17251942
]
Ahmed Hussein commented on HDFS-15679:
--
Thanks [~Jim_Brennan] for your feedback.
I am going to change the status of this Jira to "fix later" until there is a
better clarification of the changes done to the \{{close()}} interface.
> DFSOutputStream close should be a no-op when called multiple times
> --
>
> Key: HDFS-15679
> URL: https://issues.apache.org/jira/browse/HDFS-15679
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> While I was looking into the incorrect implementation of HDFS-15678, I found
> that once I implement the correct logic, the Junit test fails.
> It turns out that there is inconsistency in {{DFSOutputStream.closeImpl()}}
> introduced by HDFS-13164.
> The change in [that
> line|https://github.com/apache/hadoop/commit/51088d323359587dca7831f74c9d065c2fccc60d#diff-3a80b95578dc5079cebf0441e1dab63d5844c02fa2d04071c165ec4f7029f918R860]
> makes the close() throws exception multiple times which contradicts with
> HDFS-5335.
> That change breaks the semantic of {{close()}}. For convenience, this is a
> test code that fails because of the change in HDFS-13164.
> {code:java}
> public void testCloseTwice() throws IOException {
> DistributedFileSystem fs = cluster.getFileSystem();
> FSDataOutputStream os = fs.create(new Path("/test"));
> DFSOutputStream dos = (DFSOutputStream) Whitebox.getInternalState(os,
> "wrappedStream");
> DataStreamer streamer = (DataStreamer) Whitebox
> .getInternalState(dos, "streamer");
> @SuppressWarnings("unchecked")
> LastExceptionInStreamer ex = (LastExceptionInStreamer) Whitebox
> .getInternalState(streamer, "lastException");
> Throwable thrown = (Throwable) Whitebox.getInternalState(ex, "thrown");
> Assert.assertNull(thrown);
> // force stream to break. output stream needs to encounter a real
> // error to properly mark it closed with an exception
> cluster.shutdown(true, false);
> try {
> dos.close();
> Assert.fail("should have thrown");
> } catch (IOException e) {
> Assert.assertEquals(e.toString(), EOFException.class, e.getClass());
> }
> thrown = (Throwable) Whitebox.getInternalState(ex, "thrown");
> Assert.assertNull(thrown);
> dos.close();
> // even if the exception is set again, close should not throw it
> ex.set(new IOException("dummy"));
> dos.close();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15679) DFSOutputStream close should be a no-op when called multiple times
[
https://issues.apache.org/jira/browse/HDFS-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein resolved HDFS-15679.
--
Resolution: Later
> DFSOutputStream close should be a no-op when called multiple times
> --
>
> Key: HDFS-15679
> URL: https://issues.apache.org/jira/browse/HDFS-15679
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> While I was looking into the incorrect implementation of HDFS-15678, I found
> that once I implement the correct logic, the Junit test fails.
> It turns out that there is inconsistency in {{DFSOutputStream.closeImpl()}}
> introduced by HDFS-13164.
> The change in [that
> line|https://github.com/apache/hadoop/commit/51088d323359587dca7831f74c9d065c2fccc60d#diff-3a80b95578dc5079cebf0441e1dab63d5844c02fa2d04071c165ec4f7029f918R860]
> makes the close() throws exception multiple times which contradicts with
> HDFS-5335.
> That change breaks the semantic of {{close()}}. For convenience, this is a
> test code that fails because of the change in HDFS-13164.
> {code:java}
> public void testCloseTwice() throws IOException {
> DistributedFileSystem fs = cluster.getFileSystem();
> FSDataOutputStream os = fs.create(new Path("/test"));
> DFSOutputStream dos = (DFSOutputStream) Whitebox.getInternalState(os,
> "wrappedStream");
> DataStreamer streamer = (DataStreamer) Whitebox
> .getInternalState(dos, "streamer");
> @SuppressWarnings("unchecked")
> LastExceptionInStreamer ex = (LastExceptionInStreamer) Whitebox
> .getInternalState(streamer, "lastException");
> Throwable thrown = (Throwable) Whitebox.getInternalState(ex, "thrown");
> Assert.assertNull(thrown);
> // force stream to break. output stream needs to encounter a real
> // error to properly mark it closed with an exception
> cluster.shutdown(true, false);
> try {
> dos.close();
> Assert.fail("should have thrown");
> } catch (IOException e) {
> Assert.assertEquals(e.toString(), EOFException.class, e.getClass());
> }
> thrown = (Throwable) Whitebox.getInternalState(ex, "thrown");
> Assert.assertNull(thrown);
> dos.close();
> // even if the exception is set again, close should not throw it
> ex.set(new IOException("dummy"));
> dos.close();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15679) DFSOutputStream close should be a no-op when called multiple times
[
https://issues.apache.org/jira/browse/HDFS-15679?focusedWorklogId=526098=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-526098
]
ASF GitHub Bot logged work on HDFS-15679:
-
Author: ASF GitHub Bot
Created on: 18/Dec/20 18:53
Start Date: 18/Dec/20 18:53
Worklog Time Spent: 10m
Work Description: amahussein commented on pull request #2456:
URL: https://github.com/apache/hadoop/pull/2456#issuecomment-748259069
I will close this PR until there is a clear specification on `close()`
interface.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 526098)
Time Spent: 1.5h (was: 1h 20m)
> DFSOutputStream close should be a no-op when called multiple times
> --
>
> Key: HDFS-15679
> URL: https://issues.apache.org/jira/browse/HDFS-15679
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> While I was looking into the incorrect implementation of HDFS-15678, I found
> that once I implement the correct logic, the Junit test fails.
> It turns out that there is inconsistency in {{DFSOutputStream.closeImpl()}}
> introduced by HDFS-13164.
> The change in [that
> line|https://github.com/apache/hadoop/commit/51088d323359587dca7831f74c9d065c2fccc60d#diff-3a80b95578dc5079cebf0441e1dab63d5844c02fa2d04071c165ec4f7029f918R860]
> makes the close() throws exception multiple times which contradicts with
> HDFS-5335.
> That change breaks the semantic of {{close()}}. For convenience, this is a
> test code that fails because of the change in HDFS-13164.
> {code:java}
> public void testCloseTwice() throws IOException {
> DistributedFileSystem fs = cluster.getFileSystem();
> FSDataOutputStream os = fs.create(new Path("/test"));
> DFSOutputStream dos = (DFSOutputStream) Whitebox.getInternalState(os,
> "wrappedStream");
> DataStreamer streamer = (DataStreamer) Whitebox
> .getInternalState(dos, "streamer");
> @SuppressWarnings("unchecked")
> LastExceptionInStreamer ex = (LastExceptionInStreamer) Whitebox
> .getInternalState(streamer, "lastException");
> Throwable thrown = (Throwable) Whitebox.getInternalState(ex, "thrown");
> Assert.assertNull(thrown);
> // force stream to break. output stream needs to encounter a real
> // error to properly mark it closed with an exception
> cluster.shutdown(true, false);
> try {
> dos.close();
> Assert.fail("should have thrown");
> } catch (IOException e) {
> Assert.assertEquals(e.toString(), EOFException.class, e.getClass());
> }
> thrown = (Throwable) Whitebox.getInternalState(ex, "thrown");
> Assert.assertNull(thrown);
> dos.close();
> // even if the exception is set again, close should not throw it
> ex.set(new IOException("dummy"));
> dos.close();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15679) DFSOutputStream close should be a no-op when called multiple times
[
https://issues.apache.org/jira/browse/HDFS-15679?focusedWorklogId=526099=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-526099
]
ASF GitHub Bot logged work on HDFS-15679:
-
Author: ASF GitHub Bot
Created on: 18/Dec/20 18:53
Start Date: 18/Dec/20 18:53
Worklog Time Spent: 10m
Work Description: amahussein closed pull request #2456:
URL: https://github.com/apache/hadoop/pull/2456
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 526099)
Time Spent: 1h 40m (was: 1.5h)
> DFSOutputStream close should be a no-op when called multiple times
> --
>
> Key: HDFS-15679
> URL: https://issues.apache.org/jira/browse/HDFS-15679
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> While I was looking into the incorrect implementation of HDFS-15678, I found
> that once I implement the correct logic, the Junit test fails.
> It turns out that there is inconsistency in {{DFSOutputStream.closeImpl()}}
> introduced by HDFS-13164.
> The change in [that
> line|https://github.com/apache/hadoop/commit/51088d323359587dca7831f74c9d065c2fccc60d#diff-3a80b95578dc5079cebf0441e1dab63d5844c02fa2d04071c165ec4f7029f918R860]
> makes the close() throws exception multiple times which contradicts with
> HDFS-5335.
> That change breaks the semantic of {{close()}}. For convenience, this is a
> test code that fails because of the change in HDFS-13164.
> {code:java}
> public void testCloseTwice() throws IOException {
> DistributedFileSystem fs = cluster.getFileSystem();
> FSDataOutputStream os = fs.create(new Path("/test"));
> DFSOutputStream dos = (DFSOutputStream) Whitebox.getInternalState(os,
> "wrappedStream");
> DataStreamer streamer = (DataStreamer) Whitebox
> .getInternalState(dos, "streamer");
> @SuppressWarnings("unchecked")
> LastExceptionInStreamer ex = (LastExceptionInStreamer) Whitebox
> .getInternalState(streamer, "lastException");
> Throwable thrown = (Throwable) Whitebox.getInternalState(ex, "thrown");
> Assert.assertNull(thrown);
> // force stream to break. output stream needs to encounter a real
> // error to properly mark it closed with an exception
> cluster.shutdown(true, false);
> try {
> dos.close();
> Assert.fail("should have thrown");
> } catch (IOException e) {
> Assert.assertEquals(e.toString(), EOFException.class, e.getClass());
> }
> thrown = (Throwable) Whitebox.getInternalState(ex, "thrown");
> Assert.assertNull(thrown);
> dos.close();
> // even if the exception is set again, close should not throw it
> ex.set(new IOException("dummy"));
> dos.close();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15739) Missing Javadoc for a param in DFSNetworkTopology
[
https://issues.apache.org/jira/browse/HDFS-15739?focusedWorklogId=526097=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-526097
]
ASF GitHub Bot logged work on HDFS-15739:
-
Author: ASF GitHub Bot
Created on: 18/Dec/20 18:50
Start Date: 18/Dec/20 18:50
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2565:
URL: https://github.com/apache/hadoop/pull/2565#issuecomment-748255914
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 51s | | trunk passed |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 48s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 1s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 1m 24s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 3m 3s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 1s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 5s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 1m 5s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 39s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 10s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 53s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 3m 2s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 97m 12s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2565/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 186m 57s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2565/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2565 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 3be14723aaa4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7a88f453667 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Commented] (HDFS-15733) Add seqno in log when BlockReceiver receive packet
[
https://issues.apache.org/jira/browse/HDFS-15733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251907#comment-17251907
]
Íñigo Goiri commented on HDFS-15733:
The failed unit tests are unrelated.
+1 on [^HDFS-15733-002.patch].
> Add seqno in log when BlockReceiver receive packet
> --
>
> Key: HDFS-15733
> URL: https://issues.apache.org/jira/browse/HDFS-15733
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haibin Huang
>Assignee: Haibin Huang
>Priority: Minor
> Attachments: HDFS-15733-001.patch, HDFS-15733-002.patch
>
>
> There is a debug log show when BlockReceiver receiving a new packet, however
> now we can't tell which packet this debug log belongs to, i think it would be
> better to add a sequence number in the log.
> now the debug log like this, missing the seqno of packet
> {code:java}
> 2020-12-11,16:26:30,518 DEBUG
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving one packet for
> block BP-XXX:blk_XXX: PacketHeader with packetLen=2559 header data:
> offsetInBlock: 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12612) DFSStripedOutputStream#close will throw if called a second time with a failed streamer
[
https://issues.apache.org/jira/browse/HDFS-12612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251873#comment-17251873
]
Ahmed Hussein commented on HDFS-12612:
--
[~eddyxu], [~xiaochen], [~andrew.wang]
I was looking at the code changes of this jira. Would you mind explain why the
test method expects an exception on [closing for the second time on that
line|https://github.com/apache/hadoop/commit/f27a4ad0324aa0b4080a1c4c6bf4cd560c927e20#diff-4d4b54b7753a9843dabd2d9d1026fe9b1f7382d658c45b633cda6b71af238dbaR344]?
AFAIK, {{close()}} should be no-op for subsequent calls (which this very jira's
description is all about too). I filed a HDFS-15679 to fix that, but I was
wondering if you can shed some light on the logic behind throwing exceptions
depending for each separate scenario.
> DFSStripedOutputStream#close will throw if called a second time with a failed
> streamer
> --
>
> Key: HDFS-12612
> URL: https://issues.apache.org/jira/browse/HDFS-12612
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0-beta1
>Reporter: Andrew Wang
>Assignee: Lei (Eddy) Xu
>Priority: Major
> Labels: hdfs-ec-3.0-must-do
> Fix For: 3.0.0
>
> Attachments: HDFS-12612.00.patch, HDFS-12612.01.patch,
> HDFS-12612.02.patch, HDFS-12612.03.patch
>
>
> Found while testing with Hive. We have a cluster with 2 DNs and the XOR-2-1
> policy. If you write a file and call close() twice, it throws this exception:
> {noformat}
> 17/10/04 16:02:14 WARN hdfs.DFSOutputStream: Cannot allocate parity
> block(index=2, policy=XOR-2-1-1024k). Not enough datanodes? Exclude nodes=[]
> ...
> Caused by: java.io.IOException: Failed to get parity block, index=2
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.allocateNewBlock(DFSStripedOutputStream.java:500)
> ~[hadoop-hdfs-client-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.writeChunk(DFSStripedOutputStream.java:524)
> ~[hadoop-hdfs-client-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
> {noformat}
> This is because in DFSStripedOutputStream#closeImpl, if the stream is closed,
> we throw an exception if any of the striped streamers had an exception:
> {code}
> protected synchronized void closeImpl() throws IOException {
> if (isClosed()) {
> final MultipleIOException.Builder b = new MultipleIOException.Builder();
> for(int i = 0; i < streamers.size(); i++) {
> final StripedDataStreamer si = getStripedDataStreamer(i);
> try {
> si.getLastException().check(true);
> } catch (IOException e) {
> b.add(e);
> }
> }
> final IOException ioe = b.build();
> if (ioe != null) {
> throw ioe;
> }
> return;
> }
> {code}
> I think this is incorrect, since we only need to throw in this situation if
> we have too many failed streamers. close should also be idempotent, so it
> should throw the first time we call close if it's going to throw at all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15733) Add seqno in log when BlockReceiver receive packet
[
https://issues.apache.org/jira/browse/HDFS-15733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251849#comment-17251849
]
Hadoop QA commented on HDFS-15733:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
47s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
24s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 28s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
2s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config;
considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
1s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 40s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
|
[jira] [Reopened] (HDFS-15739) Missing Javadoc for a param in DFSNetworkTopology
[ https://issues.apache.org/jira/browse/HDFS-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhanghuazong reopened HDFS-15739: - > Missing Javadoc for a param in DFSNetworkTopology > - > > Key: HDFS-15739 > URL: https://issues.apache.org/jira/browse/HDFS-15739 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: zhanghuazong >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-15739.0.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Only add missing Javadoc for a param in method chooseRandomWithStorageType > of AppSchedulingInfo.java. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15739) Missing Javadoc for a param in DFSNetworkTopology
[ https://issues.apache.org/jira/browse/HDFS-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhanghuazong updated HDFS-15739: Summary: Missing Javadoc for a param in DFSNetworkTopology (was: Missing Javadoc for a param in org.apache.hadoop.hdfs.net.DFSNetworkTopology) > Missing Javadoc for a param in DFSNetworkTopology > - > > Key: HDFS-15739 > URL: https://issues.apache.org/jira/browse/HDFS-15739 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: zhanghuazong >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-15739.0.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Only add missing Javadoc for a param in method chooseRandomWithStorageType > of AppSchedulingInfo.java. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15739) Missing Javadoc for a param in org.apache.hadoop.hdfs.net.DFSNetworkTopology
[ https://issues.apache.org/jira/browse/HDFS-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhanghuazong resolved HDFS-15739. - Resolution: Fixed > Missing Javadoc for a param in org.apache.hadoop.hdfs.net.DFSNetworkTopology > > > Key: HDFS-15739 > URL: https://issues.apache.org/jira/browse/HDFS-15739 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: zhanghuazong >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-15739.0.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Only add missing Javadoc for a param in method chooseRandomWithStorageType > of AppSchedulingInfo.java. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15739) Missing Javadoc for a param in org.apache.hadoop.hdfs.net.DFSNetworkTopology
[ https://issues.apache.org/jira/browse/HDFS-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhanghuazong updated HDFS-15739: Attachment: HDFS-15739.0.patch > Missing Javadoc for a param in org.apache.hadoop.hdfs.net.DFSNetworkTopology > > > Key: HDFS-15739 > URL: https://issues.apache.org/jira/browse/HDFS-15739 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: zhanghuazong >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-15739.0.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Only add missing Javadoc for a param in method chooseRandomWithStorageType > of AppSchedulingInfo.java. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15739) Missing Javadoc for a param in org.apache.hadoop.hdfs.net.DFSNetworkTopology
[ https://issues.apache.org/jira/browse/HDFS-15739?focusedWorklogId=526027=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-526027 ] ASF GitHub Bot logged work on HDFS-15739: - Author: ASF GitHub Bot Created on: 18/Dec/20 15:42 Start Date: 18/Dec/20 15:42 Worklog Time Spent: 10m Work Description: langlaile1221 opened a new pull request #2565: URL: https://github.com/apache/hadoop/pull/2565 Only add missing Javadoc for a param in method chooseRandomWithStorageType of AppSchedulingInfo.java. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 526027) Remaining Estimate: 0h Time Spent: 10m > Missing Javadoc for a param in org.apache.hadoop.hdfs.net.DFSNetworkTopology > > > Key: HDFS-15739 > URL: https://issues.apache.org/jira/browse/HDFS-15739 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: zhanghuazong >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Only add missing Javadoc for a param in method chooseRandomWithStorageType > of AppSchedulingInfo.java. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15739) Missing Javadoc for a param in org.apache.hadoop.hdfs.net.DFSNetworkTopology
[ https://issues.apache.org/jira/browse/HDFS-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15739: -- Labels: pull-request-available (was: ) > Missing Javadoc for a param in org.apache.hadoop.hdfs.net.DFSNetworkTopology > > > Key: HDFS-15739 > URL: https://issues.apache.org/jira/browse/HDFS-15739 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: zhanghuazong >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Only add missing Javadoc for a param in method chooseRandomWithStorageType > of AppSchedulingInfo.java. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15739) Missing Javadoc for a param in org.apache.hadoop.hdfs.net.DFSNetworkTopology
[ https://issues.apache.org/jira/browse/HDFS-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhanghuazong updated HDFS-15739: Summary: Missing Javadoc for a param in org.apache.hadoop.hdfs.net.DFSNetworkTopology (was: Fix comment in org.apache.hadoop.hdfs.net.DFSNetworkTopology) > Missing Javadoc for a param in org.apache.hadoop.hdfs.net.DFSNetworkTopology > > > Key: HDFS-15739 > URL: https://issues.apache.org/jira/browse/HDFS-15739 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: zhanghuazong >Priority: Minor > > Only add missing Javadoc for a param in method chooseRandomWithStorageType > of AppSchedulingInfo.java. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15739) Fix comment in org.apache.hadoop.hdfs.net.DFSNetworkTopology
zhanghuazong created HDFS-15739: --- Summary: Fix comment in org.apache.hadoop.hdfs.net.DFSNetworkTopology Key: HDFS-15739 URL: https://issues.apache.org/jira/browse/HDFS-15739 Project: Hadoop HDFS Issue Type: Bug Components: hdfs Reporter: zhanghuazong Only add missing Javadoc for a param in method chooseRandomWithStorageType of AppSchedulingInfo.java. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15679) DFSOutputStream close should be a no-op when called multiple times
[
https://issues.apache.org/jira/browse/HDFS-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251824#comment-17251824
]
Jim Brennan commented on HDFS-15679:
[~eddyxu], [~andrew.wang], we would appreciate your review. We are trying to
understand why the unit tests for -HDFS-12612- expect closeImpl() to be called
twice, when it seems like the intent was to ensure it is closed only once.
> DFSOutputStream close should be a no-op when called multiple times
> --
>
> Key: HDFS-15679
> URL: https://issues.apache.org/jira/browse/HDFS-15679
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> While I was looking into the incorrect implementation of HDFS-15678, I found
> that once I implement the correct logic, the Junit test fails.
> It turns out that there is inconsistency in {{DFSOutputStream.closeImpl()}}
> introduced by HDFS-13164.
> The change in [that
> line|https://github.com/apache/hadoop/commit/51088d323359587dca7831f74c9d065c2fccc60d#diff-3a80b95578dc5079cebf0441e1dab63d5844c02fa2d04071c165ec4f7029f918R860]
> makes the close() throws exception multiple times which contradicts with
> HDFS-5335.
> That change breaks the semantic of {{close()}}. For convenience, this is a
> test code that fails because of the change in HDFS-13164.
> {code:java}
> public void testCloseTwice() throws IOException {
> DistributedFileSystem fs = cluster.getFileSystem();
> FSDataOutputStream os = fs.create(new Path("/test"));
> DFSOutputStream dos = (DFSOutputStream) Whitebox.getInternalState(os,
> "wrappedStream");
> DataStreamer streamer = (DataStreamer) Whitebox
> .getInternalState(dos, "streamer");
> @SuppressWarnings("unchecked")
> LastExceptionInStreamer ex = (LastExceptionInStreamer) Whitebox
> .getInternalState(streamer, "lastException");
> Throwable thrown = (Throwable) Whitebox.getInternalState(ex, "thrown");
> Assert.assertNull(thrown);
> // force stream to break. output stream needs to encounter a real
> // error to properly mark it closed with an exception
> cluster.shutdown(true, false);
> try {
> dos.close();
> Assert.fail("should have thrown");
> } catch (IOException e) {
> Assert.assertEquals(e.toString(), EOFException.class, e.getClass());
> }
> thrown = (Throwable) Whitebox.getInternalState(ex, "thrown");
> Assert.assertNull(thrown);
> dos.close();
> // even if the exception is set again, close should not throw it
> ex.set(new IOException("dummy"));
> dos.close();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15735) NameNode memory Leak on frequent execution of fsck
[ https://issues.apache.org/jira/browse/HDFS-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251762#comment-17251762 ] Ravuri Sushma sree commented on HDFS-15735: --- [~ayushtkn] [~John Smith] , Thank you for the reviews. I think here tracing is configurable so I opted to close it instead of removing and stopping the call of fsck may vary depending on different business requirements. > NameNode memory Leak on frequent execution of fsck > > > Key: HDFS-15735 > URL: https://issues.apache.org/jira/browse/HDFS-15735 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ravuri Sushma sree >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HDFS-15735.001.patch > > > The memory of the cluster NameNode continues to grow, and the full gc > eventually leads to the failure of the active and standby HDFS > Htrace is used to track the processing time of fsck > Checking the code it is found that the tracer object in NamenodeFsck.java was > only created but not closed because of this the memory footprint continues to > grow -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15733) Add seqno in log when BlockReceiver receive packet
[
https://issues.apache.org/jira/browse/HDFS-15733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251761#comment-17251761
]
Haibin Huang commented on HDFS-15733:
-
Thanks [~elgoiri] for comment, i have updated the patch, now the debug msg look
like this:
{code:java}
2020-12-18 21:18:10,356 DEBUG datanode.DataNode
(BlockReceiver.java:receivePacket(541)) - Receiving one packet for block
BP-985744046-192.168.0.100-1608297475921:blk_1073741828_0 seqno:100
header:PacketHeader with packetLen=8 header data: offsetInBlock: 0
{code}
> Add seqno in log when BlockReceiver receive packet
> --
>
> Key: HDFS-15733
> URL: https://issues.apache.org/jira/browse/HDFS-15733
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haibin Huang
>Assignee: Haibin Huang
>Priority: Minor
> Attachments: HDFS-15733-001.patch, HDFS-15733-002.patch
>
>
> There is a debug log show when BlockReceiver receiving a new packet, however
> now we can't tell which packet this debug log belongs to, i think it would be
> better to add a sequence number in the log.
> now the debug log like this, missing the seqno of packet
> {code:java}
> 2020-12-11,16:26:30,518 DEBUG
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving one packet for
> block BP-XXX:blk_XXX: PacketHeader with packetLen=2559 header data:
> offsetInBlock: 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15733) Add seqno in log when BlockReceiver receive packet
[
https://issues.apache.org/jira/browse/HDFS-15733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibin Huang updated HDFS-15733:
Attachment: HDFS-15733-002.patch
> Add seqno in log when BlockReceiver receive packet
> --
>
> Key: HDFS-15733
> URL: https://issues.apache.org/jira/browse/HDFS-15733
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haibin Huang
>Assignee: Haibin Huang
>Priority: Minor
> Attachments: HDFS-15733-001.patch, HDFS-15733-002.patch
>
>
> There is a debug log show when BlockReceiver receiving a new packet, however
> now we can't tell which packet this debug log belongs to, i think it would be
> better to add a sequence number in the log.
> now the debug log like this, missing the seqno of packet
> {code:java}
> 2020-12-11,16:26:30,518 DEBUG
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving one packet for
> block BP-XXX:blk_XXX: PacketHeader with packetLen=2559 header data:
> offsetInBlock: 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15737) Don't remove datanodes from outOfServiceNodeBlocks while checking in DatanodeAdminManager
[
https://issues.apache.org/jira/browse/HDFS-15737?focusedWorklogId=525954=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-525954
]
ASF GitHub Bot logged work on HDFS-15737:
-
Author: ASF GitHub Bot
Created on: 18/Dec/20 12:14
Start Date: 18/Dec/20 12:14
Worklog Time Spent: 10m
Work Description: sodonnel commented on pull request #2562:
URL: https://github.com/apache/hadoop/pull/2562#issuecomment-748053842
It looks like this same logic also exists in trunk - could you submit a
trunk PR / patch and then we can backport the change across all active branches?
I am also a little confused about this problem. The map
`outOfServiceNodeBlocks` is modified in a few places in the middle of the
Cyclic Iteration. If it threw an ConcurrentModificationException on
modification, then I would expect us to be seeing this a lot. Probably anytime
there is more than 1 node added to decommission / maintenance.
Eg, from trunk DatanodeAdminDefaultMonitor.java, here `it` is a
CyclicIterator over `outOfServiceNodeBlocks`
```
while (it.hasNext() && !exceededNumBlocksPerCheck() && namesystem
.isRunning()) {
numNodesChecked++;
final Map.Entry>
entry = it.next();
final DatanodeDescriptor dn = entry.getKey();
try {
AbstractList blocks = entry.getValue();
boolean fullScan = false;
if (dn.isMaintenance() && dn.maintenanceExpired()) {
// If maintenance expires, stop tracking it.
dnAdmin.stopMaintenance(dn);
toRemove.add(dn);
continue;
}
if (dn.isInMaintenance()) {
// The dn is IN_MAINTENANCE and the maintenance hasn't expired yet.
continue;
}
if (blocks == null) {
// This is a newly added datanode, run through its list to schedule
// under-replicated blocks for replication and collect the blocks
// that are insufficiently replicated for further tracking
LOG.debug("Newly-added node {}, doing full scan to find " +
"insufficiently-replicated blocks.", dn);
blocks = handleInsufficientlyStored(dn);
outOfServiceNodeBlocks.put(dn, blocks); // Modifies
outOfServiceNodeBlocks
...
```
Note that outOfServiceNodeBlocks is modified on the first pass, and so
`it.next()` should throw an exception on the next iteration.
Have you seen the ConcurrentModificationException logged due to this problem?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 525954)
Time Spent: 1h (was: 50m)
> Don't remove datanodes from outOfServiceNodeBlocks while checking in
> DatanodeAdminManager
> -
>
> Key: HDFS-15737
> URL: https://issues.apache.org/jira/browse/HDFS-15737
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Ye Ni
>Assignee: Ye Ni
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.10.2
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> With CyclicIteration, remove an item while iterating causes either dead loop
> or ConcurrentModificationException.
> This item should be removed by
> {{toRemove.add(dn);}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15738) Forbid the transition to Observer state when NameNode is in StartupSafeMode
[
https://issues.apache.org/jira/browse/HDFS-15738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251710#comment-17251710
]
Hadoop QA commented on HDFS-15738:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
46s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
18s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 18s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
8s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config;
considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
8s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 15s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/364/artifact/out/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red}
hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with
JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 1 new + 599 unchanged - 0
fixed = 600 total (was 599) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 3s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/364/artifact/out/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt{color}
| {color:red}
hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 1 new
+ 583 unchanged - 0 fixed = 584 total (was 583) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/364/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt{color}
| {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new +
158 unchanged -
[jira] [Updated] (HDFS-15738) Forbid the transition to Observer state when NameNode is in StartupSafeMode
[ https://issues.apache.org/jira/browse/HDFS-15738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janus Chow updated HDFS-15738: -- Attachment: HDFS-15738.001.patch Assignee: Janus Chow Status: Patch Available (was: Open) > Forbid the transition to Observer state when NameNode is in StartupSafeMode > --- > > Key: HDFS-15738 > URL: https://issues.apache.org/jira/browse/HDFS-15738 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Attachments: HDFS-15738.001.patch > > > Currently when a _getBlockLocation_ request comes to an Observer Namenode > which is in safemode, NameNode will have a check that if the result is empty, > it will reply to the client with a _RetriableException_, noting the client to > retry the request later. > And If the Observer Namenode is in startup safe mode, the client would have > to wait for the Observer NameNode to leave the safe mode. For a big cluster, > it may cause a long time of waiting for the client. In our cluster, we met > this problem, and the client needs to wait for about 30 minutes before the > service back to normal. > The reason for this situation is that the NameNode becomes the state of > Observer when it's still in safe mode getting Datanode's block reports. And > here are two solutions for this issue: > # Throw _ObserverRetryOnActiveException_ when the Observer NameNode is in > startup safe mode, redirecting the user's requests to active NN. > # Forbid the transition to Observer state when the cluster maintainer is > trying to do the transition operation. > We choose the second solution because the first one would abet the bad > operation of transition NN to Observers while it's not ready for real service. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15738) Forbid the transition to Observer state when NameNode is in StartupSafeMode
Janus Chow created HDFS-15738: - Summary: Forbid the transition to Observer state when NameNode is in StartupSafeMode Key: HDFS-15738 URL: https://issues.apache.org/jira/browse/HDFS-15738 Project: Hadoop HDFS Issue Type: Improvement Reporter: Janus Chow Currently when a _getBlockLocation_ request comes to an Observer Namenode which is in safemode, NameNode will have a check that if the result is empty, it will reply to the client with a _RetriableException_, noting the client to retry the request later. And If the Observer Namenode is in startup safe mode, the client would have to wait for the Observer NameNode to leave the safe mode. For a big cluster, it may cause a long time of waiting for the client. In our cluster, we met this problem, and the client needs to wait for about 30 minutes before the service back to normal. The reason for this situation is that the NameNode becomes the state of Observer when it's still in safe mode getting Datanode's block reports. And here are two solutions for this issue: # Throw _ObserverRetryOnActiveException_ when the Observer NameNode is in startup safe mode, redirecting the user's requests to active NN. # Forbid the transition to Observer state when the cluster maintainer is trying to do the transition operation. We choose the second solution because the first one would abet the bad operation of transition NN to Observers while it's not ready for real service. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
