[jira] [Updated] (HDFS-11150) [SPS]: Provide persistence when satisfying storage policy.
[ https://issues.apache.org/jira/browse/HDFS-11150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanbo Liu updated HDFS-11150: -- Attachment: HDFS-11150-HDFS-10285.004.patch > [SPS]: Provide persistence when satisfying storage policy. > -- > > Key: HDFS-11150 > URL: https://issues.apache.org/jira/browse/HDFS-11150 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu > Attachments: HDFS-11150-HDFS-10285.001.patch, > HDFS-11150-HDFS-10285.002.patch, HDFS-11150-HDFS-10285.003.patch, > HDFS-11150-HDFS-10285.004.patch, editsStored, editsStored.xml > > > Provide persistence for SPS in case that Hadoop cluster crashes by accident. > Basically we need to change EditLog and FsImage here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11150) [SPS]: Provide persistence when satisfying storage policy.
[
https://issues.apache.org/jira/browse/HDFS-11150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1578#comment-1578
]
Hadoop QA commented on HDFS-11150:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-11150 does not apply to HDFS-10285. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-11150 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844884/HDFS-11150-HDFS-10285.003.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17967/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [SPS]: Provide persistence when satisfying storage policy.
> --
>
> Key: HDFS-11150
> URL: https://issues.apache.org/jira/browse/HDFS-11150
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Yuanbo Liu
>Assignee: Yuanbo Liu
> Attachments: HDFS-11150-HDFS-10285.001.patch,
> HDFS-11150-HDFS-10285.002.patch, HDFS-11150-HDFS-10285.003.patch,
> editsStored, editsStored.xml
>
>
> Provide persistence for SPS in case that Hadoop cluster crashes by accident.
> Basically we need to change EditLog and FsImage here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11150) [SPS]: Provide persistence when satisfying storage policy.
[ https://issues.apache.org/jira/browse/HDFS-11150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanbo Liu updated HDFS-11150: -- Attachment: HDFS-11150-HDFS-10285.003.patch Upload a new patch to find potential test failures. I will address all your comments asap. > [SPS]: Provide persistence when satisfying storage policy. > -- > > Key: HDFS-11150 > URL: https://issues.apache.org/jira/browse/HDFS-11150 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu > Attachments: HDFS-11150-HDFS-10285.001.patch, > HDFS-11150-HDFS-10285.002.patch, HDFS-11150-HDFS-10285.003.patch, > editsStored, editsStored.xml > > > Provide persistence for SPS in case that Hadoop cluster crashes by accident. > Basically we need to change EditLog and FsImage here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11251) ConcurrentModificationException during DataNode#refreshVolumes
[
https://issues.apache.org/jira/browse/HDFS-11251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782204#comment-15782204
]
Hadoop QA commented on HDFS-11251:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 150 unchanged - 2 fixed = 151 total (was 152) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}163m 8s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}187m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11251 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844856/HDFS-11251.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ea28270ca463 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9262797 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17965/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17965/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17965/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17965/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ConcurrentModificationException during DataNode#refreshVolumes
>
[jira] [Commented] (HDFS-11251) ConcurrentModificationException during DataNode#refreshVolumes
[
https://issues.apache.org/jira/browse/HDFS-11251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782158#comment-15782158
]
Yiqun Lin commented on HDFS-11251:
--
It seems this change lead this
{code}
+ private static final int DEFAULT_STORAGES_PER_DATANODE = 512;
private MiniDFSCluster cluster;
{code}
Is that your intended change here, [~manojg]? Actually the default value {{2}}
is enough here since {{startDFSCluster(int numNameNodes, int numDataNodes)}} is
used by other tests.
> ConcurrentModificationException during DataNode#refreshVolumes
> --
>
> Key: HDFS-11251
> URL: https://issues.apache.org/jira/browse/HDFS-11251
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha2
>Reporter: Jason Lowe
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11251.01.patch, HDFS-11251.02.patch,
> HDFS-11251.03.patch
>
>
> The testAddVolumesDuringWrite case failed with a ReconfigurationException
> which appears to have been caused by a ConcurrentModificationException.
> Stacktrace details to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11252) TestFileTruncate#testTruncateWithDataNodesRestartImmediately can fail with BindException
[ https://issues.apache.org/jira/browse/HDFS-11252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782135#comment-15782135 ] Hudson commented on HDFS-11252: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11046 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11046/]) HDFS-11252. TestFileTruncate#testTruncateWithDataNodesRestartImmediately (brahma: rev 0ddb8defad6a7fd5eb69847d1789ba51952c0cf0) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java > TestFileTruncate#testTruncateWithDataNodesRestartImmediately can fail with > BindException > > > Key: HDFS-11252 > URL: https://issues.apache.org/jira/browse/HDFS-11252 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jason Lowe >Assignee: Yiqun Lin > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-11252.001.patch, HDFS-11252.002.patch > > > testTruncateWithDataNodesRestartImmediately can fail with a BindException. > The setup for TestFileTruncate has been fixed in the past to solve a bind > exception, but this is occurring after the minicluster comes up and the > datanodes are being restarted. Maybe there's a race condition there? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11274) Datanode should only check the failed volume upon IO errors
[
https://issues.apache.org/jira/browse/HDFS-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782123#comment-15782123
]
Arpit Agarwal commented on HDFS-11274:
--
The potential shutdown issue in saveDfsUsed is a nice find and fix!
Regarding the new return value of AsyncChecker#schedule (also a nice find) -
what do you think of just returning either a null or
{{Optional>}} if no disk check was scheduled? Otherwise,
a couple of comments:
# The ThrottledAsyncChecker#ScheduleResult class can be static.
# There are a couple of unchecked assignments in ThrottledAsyncChecker#schedule
(lines 110, 120 and 135).
> Datanode should only check the failed volume upon IO errors
>
>
> Key: HDFS-11274
> URL: https://issues.apache.org/jira/browse/HDFS-11274
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-11274.01.patch, HDFS-11274.02.patch,
> HDFS-11274.03.patch, HDFS-11274.04.patch
>
>
> This is a performance improvement that is possible after HDFS-11182. The goal
> is to trigger async volume check with throttling only on suspected volume
> upon datanode file IO errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11251) ConcurrentModificationException during DataNode#refreshVolumes
[
https://issues.apache.org/jira/browse/HDFS-11251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782124#comment-15782124
]
Lei (Eddy) Xu commented on HDFS-11251:
--
Hi, [~manojg]
The latest patch hangs on my local machine:
{code}
---
T E S T S
---
Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
> TEST TIMED OUT. PRINTING THREAD DUMP. <
Timestamp: 2016-12-28 01:41:15,000
"VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data34)"
daemon prio=5 tid=34502 timed_waiting
java.lang.Thread.State: TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
"pool-3452-thread-1" prio=5 tid=30550 timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"IPC Server Responder" daemon prio=5 tid=28901 runnable
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:1328)
at org.apache.hadoop.ipc.Server$Responder.run(Server.java:1311)
"VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data58)"
daemon prio=5 tid=34527 timed_waiting
java.lang.Thread.State: TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
"IPC Server handler 8 on 50718" daemon prio=5 tid=30559 timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
"VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data14)"
daemon prio=5 tid=34482 timed_waiting
java.lang.Thread.State: TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
"VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data86)"
daemon prio=5 tid=34556 timed_waiting
java.lang.Thread.State: TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
"qtp282733206-32102" daemon prio=5 tid=32102 runnable
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
[jira] [Updated] (HDFS-11252) TestFileTruncate#testTruncateWithDataNodesRestartImmediately can fail with BindException
[ https://issues.apache.org/jira/browse/HDFS-11252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11252: Affects Version/s: (was: 3.0.0-alpha2) > TestFileTruncate#testTruncateWithDataNodesRestartImmediately can fail with > BindException > > > Key: HDFS-11252 > URL: https://issues.apache.org/jira/browse/HDFS-11252 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jason Lowe >Assignee: Yiqun Lin > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-11252.001.patch, HDFS-11252.002.patch > > > testTruncateWithDataNodesRestartImmediately can fail with a BindException. > The setup for TestFileTruncate has been fixed in the past to solve a bind > exception, but this is occurring after the minicluster comes up and the > datanodes are being restarted. Maybe there's a race condition there? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11252) TestFileTruncate#testTruncateWithDataNodesRestartImmediately can fail with BindException
[ https://issues.apache.org/jira/browse/HDFS-11252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11252: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 2.8.0 Status: Resolved (was: Patch Available) Committed to trunk,branch-2 and branch-2.8..[~yiqun] thannks for your contribution. > TestFileTruncate#testTruncateWithDataNodesRestartImmediately can fail with > BindException > > > Key: HDFS-11252 > URL: https://issues.apache.org/jira/browse/HDFS-11252 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha2 >Reporter: Jason Lowe >Assignee: Yiqun Lin > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-11252.001.patch, HDFS-11252.002.patch > > > testTruncateWithDataNodesRestartImmediately can fail with a BindException. > The setup for TestFileTruncate has been fixed in the past to solve a bind > exception, but this is occurring after the minicluster comes up and the > datanodes are being restarted. Maybe there's a race condition there? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11270) Document the missing options of NameNode bootstrap command
[
https://issues.apache.org/jira/browse/HDFS-11270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782032#comment-15782032
]
Yiqun Lin commented on HDFS-11270:
--
Thanks [~liuml07] for the commit, and thanks [~brahmareddy] for the review!
> Document the missing options of NameNode bootstrap command
> --
>
> Key: HDFS-11270
> URL: https://issues.apache.org/jira/browse/HDFS-11270
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-11270.001.patch, HDFS-11270.002.patch,
> HDFS-11270.003.patch
>
>
> Currently there are missing three options introduction in {{HDFSCommands.md}}
> of command {{hdfs namenode [-bootstrapStandby]}}. Now this command can
> support {{-force}}, {{-nonInteractive}} and a additional option
> {{-skipSharedEditsCheck}}. It will be better to document what's the meaning
> of these options and that will let users know how to use it correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11274) Datanode should only check the failed volume upon IO errors
[ https://issues.apache.org/jira/browse/HDFS-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11274: -- Attachment: HDFS-11274.04.patch > Datanode should only check the failed volume upon IO errors > > > Key: HDFS-11274 > URL: https://issues.apache.org/jira/browse/HDFS-11274 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-11274.01.patch, HDFS-11274.02.patch, > HDFS-11274.03.patch, HDFS-11274.04.patch > > > This is a performance improvement that is possible after HDFS-11182. The goal > is to trigger async volume check with throttling only on suspected volume > upon datanode file IO errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11267) Avoid redefinition of storageDirs in NNStorage and cleanup its accessors in Storage
[
https://issues.apache.org/jira/browse/HDFS-11267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781957#comment-15781957
]
Lei (Eddy) Xu commented on HDFS-11267:
--
Hi, [~manojg]
Thanks a lot for cleaning the code.
I have a few questions remained:
{code}
protected List storageDirs = new
ArrayList();
{code}
Can we change it to {{private}} and adds the {{public getStorageDirs()}}?
Adding {{new ArrayList<>()}} to constructor makes duplicated code. Btw, as for
JDK 7, the second template parameter can be removed:
{code}
protected List storageDirs = new ArrayList<>();
{code}
* Regarding thread-safe iteration, the proposed patch can protected get/set the
reference of the StorageDir collection. But it might be difficult to protect
iterating the collection.
* Would using {{ArrayList()}} in {{NNStorage}} weaken the concurrency
protection of {{NNStorage}}?
Thanks!
> Avoid redefinition of storageDirs in NNStorage and cleanup its accessors in
> Storage
> ---
>
> Key: HDFS-11267
> URL: https://issues.apache.org/jira/browse/HDFS-11267
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11267.01.patch
>
>
> In the abstract class {{Storage}}, {{storageDirs}} is a protected variable
> and all its derived classes like {{NNStorage}}, {{JNStorage}},
> {{DataStorage}}.. are iterating over this non-thread safe variable without
> any proper locks. Any parallel modification operation like add or remove
> volume can mutate the backing storageDirs list and any iterators on this list
> around the same time can face {{ConcurrentModificationException}}. It would
> be good to make the variable private and restrict the access via getters and
> setters. Any thread safe restriction need to be done can then be placed on
> the parent class only.
> Also, {{NNStorage}} redefines parent class Storage's {{storageDirs}}, making
> it inconsistent with other derived classes. Would be cleaner if {{NNStorage}}
> can avoid re-defining it locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11251) ConcurrentModificationException during DataNode#refreshVolumes
[ https://issues.apache.org/jira/browse/HDFS-11251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781913#comment-15781913 ] Manoj Govindassamy commented on HDFS-11251: --- Thanks for the review [~linyiqun]. Updated the patch with above changes. > ConcurrentModificationException during DataNode#refreshVolumes > -- > > Key: HDFS-11251 > URL: https://issues.apache.org/jira/browse/HDFS-11251 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha2 >Reporter: Jason Lowe >Assignee: Manoj Govindassamy > Attachments: HDFS-11251.01.patch, HDFS-11251.02.patch, > HDFS-11251.03.patch > > > The testAddVolumesDuringWrite case failed with a ReconfigurationException > which appears to have been caused by a ConcurrentModificationException. > Stacktrace details to follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11251) ConcurrentModificationException during DataNode#refreshVolumes
[ https://issues.apache.org/jira/browse/HDFS-11251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj Govindassamy updated HDFS-11251: -- Attachment: HDFS-11251.03.patch > ConcurrentModificationException during DataNode#refreshVolumes > -- > > Key: HDFS-11251 > URL: https://issues.apache.org/jira/browse/HDFS-11251 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha2 >Reporter: Jason Lowe >Assignee: Manoj Govindassamy > Attachments: HDFS-11251.01.patch, HDFS-11251.02.patch, > HDFS-11251.03.patch > > > The testAddVolumesDuringWrite case failed with a ReconfigurationException > which appears to have been caused by a ConcurrentModificationException. > Stacktrace details to follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11186) [SPS]: Daemon thread of SPS should start only in Active NN
[
https://issues.apache.org/jira/browse/HDFS-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781900#comment-15781900
]
Wei Zhou commented on HDFS-11186:
-
This is a good way to bypass the issue, but I had add codes to other tests in
order to make them to work with default true in HDFS-10885, and also
{{TestStoragePolicySatisfier}} thinks it should be true by default, so a lot of
modifications should be made when change the value to false. What's your
opinion?
> [SPS]: Daemon thread of SPS should start only in Active NN
> --
>
> Key: HDFS-11186
> URL: https://issues.apache.org/jira/browse/HDFS-11186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Attachments: HDFS-11186-HDFS-10285.00.patch,
> HDFS-11186-HDFS-10285.01.patch, HDFS-11186-HDFS-10285.02.patch
>
>
> As discussed in [HDFS-10885
> |https://issues.apache.org/jira/browse/HDFS-10885], we need to ensure that
> SPS is started only in Active NN. This JIRA is opened for discussion and
> tracking.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11121) Add assertions to BlockInfo#addStorage to protect from breaking reportedBlock-blockGroup mapping
[
https://issues.apache.org/jira/browse/HDFS-11121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781883#comment-15781883
]
Hadoop QA commented on HDFS-11121:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 10 unchanged - 3 fixed = 10 total (was 13) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 67m
18s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11121 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844849/HDFS-11121.2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 493c511aba63 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9262797 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17964/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17964/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add assertions to BlockInfo#addStorage to protect from breaking
> reportedBlock-blockGroup mapping
>
>
> Key: HDFS-11121
> URL: https://issues.apache.org/jira/browse/HDFS-11121
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Critical
[jira] [Commented] (HDFS-11186) [SPS]: Daemon thread of SPS should start only in Active NN
[
https://issues.apache.org/jira/browse/HDFS-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781867#comment-15781867
]
Wei Zhou commented on HDFS-11186:
-
Sorry, {{In order to trigger Jenkins to work}} this comment is not related to
your question at all. I didn't see your comments then and just wanted to make
the patch rebuild. Please ignore this comment.
> [SPS]: Daemon thread of SPS should start only in Active NN
> --
>
> Key: HDFS-11186
> URL: https://issues.apache.org/jira/browse/HDFS-11186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Attachments: HDFS-11186-HDFS-10285.00.patch,
> HDFS-11186-HDFS-10285.01.patch, HDFS-11186-HDFS-10285.02.patch
>
>
> As discussed in [HDFS-10885
> |https://issues.apache.org/jira/browse/HDFS-10885], we need to ensure that
> SPS is started only in Active NN. This JIRA is opened for discussion and
> tracking.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11186) [SPS]: Daemon thread of SPS should start only in Active NN
[ https://issues.apache.org/jira/browse/HDFS-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781861#comment-15781861 ] Rakesh R commented on HDFS-11186: - OK, good catch. It looks like this thread.join timeout was added by another jira. Probably, we could keep an eye on Jenkins failures. Do you think to keep DFS_STORAGE_POLICY_SATISFIER_ACTIVATE_DEFAULT to false to make Jenkins happy? > [SPS]: Daemon thread of SPS should start only in Active NN > -- > > Key: HDFS-11186 > URL: https://issues.apache.org/jira/browse/HDFS-11186 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Wei Zhou >Assignee: Wei Zhou > Attachments: HDFS-11186-HDFS-10285.00.patch, > HDFS-11186-HDFS-10285.01.patch, HDFS-11186-HDFS-10285.02.patch > > > As discussed in [HDFS-10885 > |https://issues.apache.org/jira/browse/HDFS-10885], we need to ensure that > SPS is started only in Active NN. This JIRA is opened for discussion and > tracking. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10899) Add functionality to re-encrypt EDEKs.
[
https://issues.apache.org/jira/browse/HDFS-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781846#comment-15781846
]
Hadoop QA commented on HDFS-10899:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 52s{color} | {color:orange} hadoop-hdfs-project: The patch generated 14 new
+ 1571 unchanged - 2 fixed = 1585 total (was 1573) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 38s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.TestDataTransferKeepalive |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10899 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844842/HDFS-10899.02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc xml |
| uname | Linux a61e906a21d4 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9262797 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Issue Comment Deleted] (HDFS-11186) [SPS]: Daemon thread of SPS should start only in Active NN
[
https://issues.apache.org/jira/browse/HDFS-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zhou updated HDFS-11186:
Comment: was deleted
(was: Hi [~rakeshr], thanks for reviewing the patch! The reason is that it
takes extra time to stop NN than previous.
{code}
try {
storagePolicySatisfierThread.join(3000);
} catch (InterruptedException ie) {
}
{code}
It will take longer time to finish for tests that have to change NN HA states
or restart NN, and this leads to timeout failure.)
> [SPS]: Daemon thread of SPS should start only in Active NN
> --
>
> Key: HDFS-11186
> URL: https://issues.apache.org/jira/browse/HDFS-11186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Attachments: HDFS-11186-HDFS-10285.00.patch,
> HDFS-11186-HDFS-10285.01.patch, HDFS-11186-HDFS-10285.02.patch
>
>
> As discussed in [HDFS-10885
> |https://issues.apache.org/jira/browse/HDFS-10885], we need to ensure that
> SPS is started only in Active NN. This JIRA is opened for discussion and
> tracking.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11186) [SPS]: Daemon thread of SPS should start only in Active NN
[
https://issues.apache.org/jira/browse/HDFS-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781841#comment-15781841
]
Wei Zhou commented on HDFS-11186:
-
Hi [~rakeshr], thanks for reviewing the patch! The reason is that it takes
extra time to stop NN than previous.
{code}
try {
storagePolicySatisfierThread.join(3000);
} catch (InterruptedException ie) {
}
{code}
It will take longer time to finish for tests that have to change NN HA states
or restart NN, and this leads to timeout failure.
> [SPS]: Daemon thread of SPS should start only in Active NN
> --
>
> Key: HDFS-11186
> URL: https://issues.apache.org/jira/browse/HDFS-11186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Attachments: HDFS-11186-HDFS-10285.00.patch,
> HDFS-11186-HDFS-10285.01.patch, HDFS-11186-HDFS-10285.02.patch
>
>
> As discussed in [HDFS-10885
> |https://issues.apache.org/jira/browse/HDFS-10885], we need to ensure that
> SPS is started only in Active NN. This JIRA is opened for discussion and
> tracking.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11186) [SPS]: Daemon thread of SPS should start only in Active NN
[
https://issues.apache.org/jira/browse/HDFS-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781840#comment-15781840
]
Wei Zhou commented on HDFS-11186:
-
Hi [~rakeshr], thanks for reviewing the patch! The reason is that it takes
extra time to stop NN than previous.
{code}
try {
storagePolicySatisfierThread.join(3000);
} catch (InterruptedException ie) {
}
{code}
It will take longer time to finish for tests that have to change NN HA states
or restart NN, and this leads to timeout failure.
> [SPS]: Daemon thread of SPS should start only in Active NN
> --
>
> Key: HDFS-11186
> URL: https://issues.apache.org/jira/browse/HDFS-11186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Attachments: HDFS-11186-HDFS-10285.00.patch,
> HDFS-11186-HDFS-10285.01.patch, HDFS-11186-HDFS-10285.02.patch
>
>
> As discussed in [HDFS-10885
> |https://issues.apache.org/jira/browse/HDFS-10885], we need to ensure that
> SPS is started only in Active NN. This JIRA is opened for discussion and
> tracking.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11251) ConcurrentModificationException during DataNode#refreshVolumes
[
https://issues.apache.org/jira/browse/HDFS-11251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781823#comment-15781823
]
Yiqun Lin edited comment on HDFS-11251 at 12/28/16 2:21 AM:
Thanks [~manojg] for updating the patch. The latest patch looks pretty good
now. Two minor comments:
* Can we define a var named {{DEFAULT_STORAGES_PER_DATANODE}} to replace {{2}}?
That will be easily understood.
{code}
private void startDFSCluster(int numNameNodes, int numDataNodes)
throws IOException {
+startDFSCluster(numNameNodes, numDataNodes, 2);
+ }
{code}
* The delay time of {{addVolume}} is a little short. I tested your patch in my
local many times, the most of the results were still passed with the
{{ArrayList}}.
{code}
if (r.nextInt(10) > 4) {
int s = r.nextInt(10) + 1;
Thread.sleep(s);
}
{code}
I increased the delay here, change {{Thread.sleep(s)}} to {{Thread.sleep(s *
100)}}, then the tests runs as we expected,
+1 once these are addressed. Thanks.
was (Author: linyiqun):
Thanks [~manojg] for updating the patch. The latest patch looks pretty good
now. Two minor comments:
* Can we define a var named {{DEFAULT_STORAGES_PER_DATANODE}} to replace {{2}}?
That will be easily understood.
{quote}
private void startDFSCluster(int numNameNodes, int numDataNodes)
throws IOException {
+startDFSCluster(numNameNodes, numDataNodes, 2);
+ }
{quote}
* The delay time of {{addVolume}} is a little short. I tested your patch in my
local many times, the most of the results were still passed with the
{{ArrayList}}.
{quote}
if (r.nextInt(10) > 4) {
int s = r.nextInt(10) + 1;
Thread.sleep(s);
}
{quote}
I increased the delay here, change {{Thread.sleep(s)}} to {{Thread.sleep(s *
100)}}, then the tests runs as we expected,
+1 once these are addressed. Thanks.
> ConcurrentModificationException during DataNode#refreshVolumes
> --
>
> Key: HDFS-11251
> URL: https://issues.apache.org/jira/browse/HDFS-11251
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha2
>Reporter: Jason Lowe
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11251.01.patch, HDFS-11251.02.patch
>
>
> The testAddVolumesDuringWrite case failed with a ReconfigurationException
> which appears to have been caused by a ConcurrentModificationException.
> Stacktrace details to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11186) [SPS]: Daemon thread of SPS should start only in Active NN
[ https://issues.apache.org/jira/browse/HDFS-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781818#comment-15781818 ] Rakesh R edited comment on HDFS-11186 at 12/28/16 2:20 AM: --- Do you meant, that the current patch has increased the testing time for these test cases? was (Author: rakeshr): Do you meant, that the current patch is causing the testing time for these test cases? > [SPS]: Daemon thread of SPS should start only in Active NN > -- > > Key: HDFS-11186 > URL: https://issues.apache.org/jira/browse/HDFS-11186 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Wei Zhou >Assignee: Wei Zhou > Attachments: HDFS-11186-HDFS-10285.00.patch, > HDFS-11186-HDFS-10285.01.patch, HDFS-11186-HDFS-10285.02.patch > > > As discussed in [HDFS-10885 > |https://issues.apache.org/jira/browse/HDFS-10885], we need to ensure that > SPS is started only in Active NN. This JIRA is opened for discussion and > tracking. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11251) ConcurrentModificationException during DataNode#refreshVolumes
[
https://issues.apache.org/jira/browse/HDFS-11251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781823#comment-15781823
]
Yiqun Lin commented on HDFS-11251:
--
Thanks [~manojg] for updating the patch. The latest patch looks pretty good
now. Two minor comments:
* Can we define a var named {{DEFAULT_STORAGES_PER_DATANODE}} to replace {{2}}?
That will be easily understood.
{quote}
private void startDFSCluster(int numNameNodes, int numDataNodes)
throws IOException {
+startDFSCluster(numNameNodes, numDataNodes, 2);
+ }
{quote}
* The delay time of {{addVolume}} is a little short. I tested your patch in my
local many times, the most of the results were still passed with the
{{ArrayList}}.
{quote}
if (r.nextInt(10) > 4) {
int s = r.nextInt(10) + 1;
Thread.sleep(s);
}
{quote}
I increased the delay here, change {{Thread.sleep(s)}} to {{Thread.sleep(s *
100)}}, then the tests runs as we expected,
+1 once these are addressed. Thanks.
> ConcurrentModificationException during DataNode#refreshVolumes
> --
>
> Key: HDFS-11251
> URL: https://issues.apache.org/jira/browse/HDFS-11251
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha2
>Reporter: Jason Lowe
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11251.01.patch, HDFS-11251.02.patch
>
>
> The testAddVolumesDuringWrite case failed with a ReconfigurationException
> which appears to have been caused by a ConcurrentModificationException.
> Stacktrace details to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11186) [SPS]: Daemon thread of SPS should start only in Active NN
[ https://issues.apache.org/jira/browse/HDFS-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781818#comment-15781818 ] Rakesh R commented on HDFS-11186: - Do you meant, that the current patch is causing the testing time for these test cases? > [SPS]: Daemon thread of SPS should start only in Active NN > -- > > Key: HDFS-11186 > URL: https://issues.apache.org/jira/browse/HDFS-11186 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Wei Zhou >Assignee: Wei Zhou > Attachments: HDFS-11186-HDFS-10285.00.patch, > HDFS-11186-HDFS-10285.01.patch, HDFS-11186-HDFS-10285.02.patch > > > As discussed in [HDFS-10885 > |https://issues.apache.org/jira/browse/HDFS-10885], we need to ensure that > SPS is started only in Active NN. This JIRA is opened for discussion and > tracking. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11186) [SPS]: Daemon thread of SPS should start only in Active NN
[
https://issues.apache.org/jira/browse/HDFS-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781804#comment-15781804
]
Rakesh R edited comment on HDFS-11186 at 12/28/16 2:14 AM:
---
Thanks [~zhouwei] for the updates. Overall the patch looks good to me. Could
you clarify on the below test changes in your patch,
{{TestHAStateTransitions.java}}, {{TestSeveralNameNodes.java}} - any specific
reason to increase to a high value?
was (Author: rakeshr):
Thanks [~zhouwei] for the updates. Overall the patch looks good to me. Could
you clarify on the below test changes in your patch,
- {{TestHAStateTransitions.java}}, {{TestSeveralNameNodes.java}} - any specific
reason to increase to a high value?
- {{TestStoragePolicySatisfierWithHA}}- can we keep @Test(timeout = 3) for
TestStoragePolicySatisfierWithHA#testWhenNNHAStateChanges?
> [SPS]: Daemon thread of SPS should start only in Active NN
> --
>
> Key: HDFS-11186
> URL: https://issues.apache.org/jira/browse/HDFS-11186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Attachments: HDFS-11186-HDFS-10285.00.patch,
> HDFS-11186-HDFS-10285.01.patch, HDFS-11186-HDFS-10285.02.patch
>
>
> As discussed in [HDFS-10885
> |https://issues.apache.org/jira/browse/HDFS-10885], we need to ensure that
> SPS is started only in Active NN. This JIRA is opened for discussion and
> tracking.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11186) [SPS]: Daemon thread of SPS should start only in Active NN
[ https://issues.apache.org/jira/browse/HDFS-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zhou updated HDFS-11186: Status: Open (was: Patch Available) In order to trigger Jenkins to work. > [SPS]: Daemon thread of SPS should start only in Active NN > -- > > Key: HDFS-11186 > URL: https://issues.apache.org/jira/browse/HDFS-11186 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Wei Zhou >Assignee: Wei Zhou > Attachments: HDFS-11186-HDFS-10285.00.patch, > HDFS-11186-HDFS-10285.01.patch, HDFS-11186-HDFS-10285.02.patch > > > As discussed in [HDFS-10885 > |https://issues.apache.org/jira/browse/HDFS-10885], we need to ensure that > SPS is started only in Active NN. This JIRA is opened for discussion and > tracking. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11186) [SPS]: Daemon thread of SPS should start only in Active NN
[ https://issues.apache.org/jira/browse/HDFS-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zhou updated HDFS-11186: Status: Patch Available (was: Open) > [SPS]: Daemon thread of SPS should start only in Active NN > -- > > Key: HDFS-11186 > URL: https://issues.apache.org/jira/browse/HDFS-11186 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Wei Zhou >Assignee: Wei Zhou > Attachments: HDFS-11186-HDFS-10285.00.patch, > HDFS-11186-HDFS-10285.01.patch, HDFS-11186-HDFS-10285.02.patch > > > As discussed in [HDFS-10885 > |https://issues.apache.org/jira/browse/HDFS-10885], we need to ensure that > SPS is started only in Active NN. This JIRA is opened for discussion and > tracking. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11186) [SPS]: Daemon thread of SPS should start only in Active NN
[
https://issues.apache.org/jira/browse/HDFS-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781804#comment-15781804
]
Rakesh R commented on HDFS-11186:
-
Thanks [~zhouwei] for the updates. Overall the patch looks good to me. Could
you clarify on the below test changes in your patch,
- {{TestHAStateTransitions.java}}, {{TestSeveralNameNodes.java}} - any specific
reason to increase to a high value?
- {{TestStoragePolicySatisfierWithHA}}- can we keep @Test(timeout = 3) for
TestStoragePolicySatisfierWithHA#testWhenNNHAStateChanges?
> [SPS]: Daemon thread of SPS should start only in Active NN
> --
>
> Key: HDFS-11186
> URL: https://issues.apache.org/jira/browse/HDFS-11186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Attachments: HDFS-11186-HDFS-10285.00.patch,
> HDFS-11186-HDFS-10285.01.patch, HDFS-11186-HDFS-10285.02.patch
>
>
> As discussed in [HDFS-10885
> |https://issues.apache.org/jira/browse/HDFS-10885], we need to ensure that
> SPS is started only in Active NN. This JIRA is opened for discussion and
> tracking.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11274) Datanode should only check the failed volume upon IO errors
[
https://issues.apache.org/jira/browse/HDFS-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781787#comment-15781787
]
Hadoop QA commented on HDFS-11274:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 29s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 7 new + 391 unchanged - 6 fixed = 398 total (was 397) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}121m 15s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}147m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestModTime |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.server.datanode.checker.TestDatasetVolumeChecker |
| Timed out junit tests | org.apache.hadoop.fs.viewfs.TestViewFileSystemHdfs |
| | org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
|
| | org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool |
| | org.apache.hadoop.hdfs.server.datanode.TestDataNodeExit |
| | org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | org.apache.hadoop.fs.viewfs.TestViewFileSystemWithAcls |
| | org.apache.hadoop.hdfs.server.namenode.TestFsckWithMultipleNameNodes |
| | org.apache.hadoop.hdfs.server.namenode.TestCheckpoint |
| | org.apache.hadoop.fs.viewfs.TestViewFileSystemWithXAttrs |
| |
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
| | org.apache.hadoop.fs.viewfs.TestViewFsWithAcls |
| | org.apache.hadoop.fs.viewfs.TestViewFsWithXAttrs |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11274 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844835/HDFS-11274.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux daf68da48981 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Updated] (HDFS-11267) Avoid redefinition of storageDirs in NNStorage and cleanup its accessors in Storage
[
https://issues.apache.org/jira/browse/HDFS-11267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11267:
--
Attachment: HDFS-11267.01.patch
Attaching v01 patch to address the following:
1. Avoid storageDirs getting re-defined in NNStorage. The redefinition is not
needed as HDFS-11251 is anyway making the parent member variable stroageDir a
{{CopyOnWriteArrayList}}
2. Make {{Storage#storageDirs}} a private member and expose get() method for
the same.
3. Clean up all direct accessors of storageDirs with getStorageDirs() method
[~eddyxu], can you please take a look at the patch ?
> Avoid redefinition of storageDirs in NNStorage and cleanup its accessors in
> Storage
> ---
>
> Key: HDFS-11267
> URL: https://issues.apache.org/jira/browse/HDFS-11267
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11267.01.patch
>
>
> In the abstract class {{Storage}}, {{storageDirs}} is a protected variable
> and all its derived classes like {{NNStorage}}, {{JNStorage}},
> {{DataStorage}}.. are iterating over this non-thread safe variable without
> any proper locks. Any parallel modification operation like add or remove
> volume can mutate the backing storageDirs list and any iterators on this list
> around the same time can face {{ConcurrentModificationException}}. It would
> be good to make the variable private and restrict the access via getters and
> setters. Any thread safe restriction need to be done can then be placed on
> the parent class only.
> Also, {{NNStorage}} redefines parent class Storage's {{storageDirs}}, making
> it inconsistent with other derived classes. Would be cleaner if {{NNStorage}}
> can avoid re-defining it locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11267) Avoid redefinition of storageDirs in NNStorage and cleanup its accessors in Storage
[
https://issues.apache.org/jira/browse/HDFS-11267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781739#comment-15781739
]
Manoj Govindassamy commented on HDFS-11267:
---
The fix for HDFS-11251 (ConcurrentModificationException during
DataNode#refreshVolumes) making the {{Storage#storageDirs}} a
{{CopyOnWriteArrayList}} is needed for this bug. On top of HDFS-11251 fix, we
need more clean up patches.
> Avoid redefinition of storageDirs in NNStorage and cleanup its accessors in
> Storage
> ---
>
> Key: HDFS-11267
> URL: https://issues.apache.org/jira/browse/HDFS-11267
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>
> In the abstract class {{Storage}}, {{storageDirs}} is a protected variable
> and all its derived classes like {{NNStorage}}, {{JNStorage}},
> {{DataStorage}}.. are iterating over this non-thread safe variable without
> any proper locks. Any parallel modification operation like add or remove
> volume can mutate the backing storageDirs list and any iterators on this list
> around the same time can face {{ConcurrentModificationException}}. It would
> be good to make the variable private and restrict the access via getters and
> setters. Any thread safe restriction need to be done can then be placed on
> the parent class only.
> Also, {{NNStorage}} redefines parent class Storage's {{storageDirs}}, making
> it inconsistent with other derived classes. Would be cleaner if {{NNStorage}}
> can avoid re-defining it locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11121) Add assertions to BlockInfo#addStorage to protect from breaking reportedBlock-blockGroup mapping
[
https://issues.apache.org/jira/browse/HDFS-11121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-11121:
Attachment: HDFS-11121.2.patch
I rebased the last patch with the latest trunk and uploaded it.
> Add assertions to BlockInfo#addStorage to protect from breaking
> reportedBlock-blockGroup mapping
>
>
> Key: HDFS-11121
> URL: https://issues.apache.org/jira/browse/HDFS-11121
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Critical
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-11121.1.patch, HDFS-11121.2.patch
>
>
> There are not any assertions in {{BlockInfo.addStorage}}. This may cause that
> {{BlockInfo}} instances accept strange block reports and result in serious
> bugs, like HDFS-10858.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11121) Add assertions to BlockInfo#addStorage to protect from breaking reportedBlock-blockGroup mapping
[
https://issues.apache.org/jira/browse/HDFS-11121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781732#comment-15781732
]
Takanobu Asanuma commented on HDFS-11121:
-
Thanks for your thoughts. I agree with you. So let's leave the constructor of
{{BlockInfoStriped}} as is for now.
> Add assertions to BlockInfo#addStorage to protect from breaking
> reportedBlock-blockGroup mapping
>
>
> Key: HDFS-11121
> URL: https://issues.apache.org/jira/browse/HDFS-11121
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Critical
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-11121.1.patch
>
>
> There are not any assertions in {{BlockInfo.addStorage}}. This may cause that
> {{BlockInfo}} instances accept strange block reports and result in serious
> bugs, like HDFS-10858.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10899) Add functionality to re-encrypt EDEKs.
[ https://issues.apache.org/jira/browse/HDFS-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-10899: - Attachment: HDFS-10899.02.patch > Add functionality to re-encrypt EDEKs. > -- > > Key: HDFS-10899 > URL: https://issues.apache.org/jira/browse/HDFS-10899 > Project: Hadoop HDFS > Issue Type: New Feature > Components: encryption, kms >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-10899.01.patch, HDFS-10899.02.patch, > HDFS-10899.02.patch, HDFS-10899.wip.2.patch, HDFS-10899.wip.patch, Re-encrypt > edek design doc.pdf > > > Currently when an encryption zone (EZ) key is rotated, it only takes effect > on new EDEKs. We should provide a way to re-encrypt EDEKs after the EZ key > rotation, for improved security. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10899) Add functionality to re-encrypt EDEKs.
[ https://issues.apache.org/jira/browse/HDFS-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-10899: - Attachment: (was: HDFS-10899.02.patch) > Add functionality to re-encrypt EDEKs. > -- > > Key: HDFS-10899 > URL: https://issues.apache.org/jira/browse/HDFS-10899 > Project: Hadoop HDFS > Issue Type: New Feature > Components: encryption, kms >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-10899.01.patch, HDFS-10899.02.patch, > HDFS-10899.wip.2.patch, HDFS-10899.wip.patch, Re-encrypt edek design doc.pdf > > > Currently when an encryption zone (EZ) key is rotated, it only takes effect > on new EDEKs. We should provide a way to re-encrypt EDEKs after the EZ key > rotation, for improved security. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10899) Add functionality to re-encrypt EDEKs.
[
https://issues.apache.org/jira/browse/HDFS-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781619#comment-15781619
]
Hadoop QA commented on HDFS-10899:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 47s{color} | {color:orange} hadoop-hdfs-project: The patch generated 29 new
+ 1529 unchanged - 2 fixed = 1558 total (was 1531) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
54s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 15s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}101m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
|
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.TestDFSInotifyEventInputStream |
| | hadoop.hdfs.server.namenode.TestCheckpoint |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10899 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844832/HDFS-10899.02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc xml |
| uname | Linux 799095a23714 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk /
[jira] [Commented] (HDFS-9391) Update webUI/JMX to display maintenance state info
[
https://issues.apache.org/jira/browse/HDFS-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781584#comment-15781584
]
Manoj Govindassamy commented on HDFS-9391:
--
[~eddyxu], [~mingma],
Are you ok leaving "decommissioned" attr in {{FSNameSystem#getDeadNodes()}}
even after the addition of new "adminState", so as to preserve compatibility ?
And, any other review comments ? Please let me know.
> Update webUI/JMX to display maintenance state info
> --
>
> Key: HDFS-9391
> URL: https://issues.apache.org/jira/browse/HDFS-9391
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Ming Ma
>Assignee: Manoj Govindassamy
> Attachments: HDFS-9391-MaintenanceMode-WebUI.pdf, HDFS-9391.01.patch,
> HDFS-9391.02.patch, Maintenance webUI.png
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11251) ConcurrentModificationException during DataNode#refreshVolumes
[
https://issues.apache.org/jira/browse/HDFS-11251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11251:
--
Attachment: HDFS-11251.02.patch
Attached v02 patch to address the following:
1. Added a new test {{TestDataNodeHotSwapVolumes#testAddVolumesConcurrently}}
to recreate the problem of ConcurrentModificationException during addVolume and
fail when run without the proposed fix.
2. HDFS-11267 has been filed to track the checkstyle issue.
PS: testAddVolumesConcurrently currently has 40 new volumes to be added
concurrently. Without fix, this test fails most of the times. You might want to
increase the volume count to a higher number to make it fail all the times.
[~eddyxu], [~linyiqun], can you please take a look at the patch ?
> ConcurrentModificationException during DataNode#refreshVolumes
> --
>
> Key: HDFS-11251
> URL: https://issues.apache.org/jira/browse/HDFS-11251
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha2
>Reporter: Jason Lowe
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11251.01.patch, HDFS-11251.02.patch
>
>
> The testAddVolumesDuringWrite case failed with a ReconfigurationException
> which appears to have been caused by a ConcurrentModificationException.
> Stacktrace details to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11274) Datanode should only check the failed volume upon IO errors
[ https://issues.apache.org/jira/browse/HDFS-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781508#comment-15781508 ] Xiaoyu Yao edited comment on HDFS-11274 at 12/27/16 11:49 PM: -- Delta from patch 02: 1. Add a ScheduleResult interface to get the ListenableFuture and whether a new target check is added to ThrottledAsyncChecker. This way, the async callback will be called once per scheduled check. 2. Avoid cyclic volume check during volume check result handler, more specifically the BlockPoolSlice#saveDfsUsed() during volume shutdown. 3. Fix the unit test as the volume failure counter can be larger due to the async volume check and the fact that min-check-gap is set as 0 during certain test. was (Author: xyao): Delta from patch 02: 1. Add a ScheduleResult interface to get the ListenableFuture and whether a new target check is added to ThrottledAsyncChecker. 2. Avoid cyclic volume check during volume check result handler. 3. Fix the unit test. > Datanode should only check the failed volume upon IO errors > > > Key: HDFS-11274 > URL: https://issues.apache.org/jira/browse/HDFS-11274 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-11274.01.patch, HDFS-11274.02.patch, > HDFS-11274.03.patch > > > This is a performance improvement that is possible after HDFS-11182. The goal > is to trigger async volume check with throttling only on suspected volume > upon datanode file IO errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11274) Datanode should only check the failed volume upon IO errors
[ https://issues.apache.org/jira/browse/HDFS-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11274: -- Attachment: HDFS-11274.03.patch Delta from patch 02: 1. Add a ScheduleResult interface to get the ListenableFuture and whether a new target check is added to ThrottledAsyncChecker. 2. Avoid cyclic volume check during volume check result handler. 3. Fix the unit test. > Datanode should only check the failed volume upon IO errors > > > Key: HDFS-11274 > URL: https://issues.apache.org/jira/browse/HDFS-11274 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-11274.01.patch, HDFS-11274.02.patch, > HDFS-11274.03.patch > > > This is a performance improvement that is possible after HDFS-11182. The goal > is to trigger async volume check with throttling only on suspected volume > upon datanode file IO errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11239) [SPS]: Check Mover file ID lease also to determine whether Mover is running
[
https://issues.apache.org/jira/browse/HDFS-11239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781485#comment-15781485
]
Uma Maheswara Rao G commented on HDFS-11239:
# .
{code}
+ public boolean isFileOpenedForWrite(String path) {
{code}
Can you please add @Override flag?
# .
{code}
@Test(timeout = 2)
+ @Test(timeout = 12)
public void testClusterIdMismatchAtStartupWithHA() throws Exception {
{code}
Is this failure related to this patch? if No, could you please file separate
JIRA for that failure? This is to avoid unrelated changes to this patch.
> [SPS]: Check Mover file ID lease also to determine whether Mover is running
> ---
>
> Key: HDFS-11239
> URL: https://issues.apache.org/jira/browse/HDFS-11239
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Attachments: HDFS-11239-HDFS-10285.00.patch,
> HDFS-11239-HDFS-10285.01.patch, HDFS-11239-HDFS-10285.02.patch
>
>
> Currently in SPS only checks the Mover ID file existence to determine whether
> a Mover is running, this can be an issue when Mover exists unexpected without
> deleting the ID file, and this further stops SPS to function. This is a
> following on to HDFS-10885 and there we bypassed this due to some
> implementation problems. This issue can be fixed after HDFS-11123.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10899) Add functionality to re-encrypt EDEKs.
[ https://issues.apache.org/jira/browse/HDFS-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-10899: - Attachment: HDFS-10899.02.patch Rebased on latest trunk, and fixed precommits. > Add functionality to re-encrypt EDEKs. > -- > > Key: HDFS-10899 > URL: https://issues.apache.org/jira/browse/HDFS-10899 > Project: Hadoop HDFS > Issue Type: New Feature > Components: encryption, kms >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-10899.01.patch, HDFS-10899.02.patch, > HDFS-10899.wip.2.patch, HDFS-10899.wip.patch, Re-encrypt edek design doc.pdf > > > Currently when an encryption zone (EZ) key is rotated, it only takes effect > on new EDEKs. We should provide a way to re-encrypt EDEKs after the EZ key > rotation, for improved security. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10759) Change fsimage bool isStriped from boolean to an enum
[
https://issues.apache.org/jira/browse/HDFS-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781391#comment-15781391
]
Wei-Chiu Chuang commented on HDFS-10759:
I like the proposal and retriggered the Jenkins build for patch 0002, but it
doesn't apply any more. [~ehiggs] would you rebase your patch? Thanks
> Change fsimage bool isStriped from boolean to an enum
> -
>
> Key: HDFS-10759
> URL: https://issues.apache.org/jira/browse/HDFS-10759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0-alpha1, 3.0.0-beta1, 3.0.0-alpha2
>Reporter: Ewan Higgs
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10759.0001.patch, HDFS-10759.0002.patch
>
>
> The new erasure coding project has updated the protocol for fsimage such that
> the {{INodeFile}} has a boolean '{{isStriped}}'. I think this is better as an
> enum or integer since a boolean precludes any future block types.
> For example:
> {code}
> enum BlockType {
> CONTIGUOUS = 0,
> STRIPED = 1,
> }
> {code}
> We can also make this more robust to future changes where there are different
> block types supported in a staged rollout. Here, we would use
> {{UNKNOWN_BLOCK_TYPE}} as the first value since this is the default value.
> See
> [here|http://androiddevblog.com/protocol-buffers-pitfall-adding-enum-values/]
> for more discussion.
> {code}
> enum BlockType {
> UNKNOWN_BLOCK_TYPE = 0,
> CONTIGUOUS = 1,
> STRIPED = 2,
> }
> {code}
> But I'm not convinced this is necessary since there are other enums that
> don't use this approach.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10759) Change fsimage bool isStriped from boolean to an enum
[
https://issues.apache.org/jira/browse/HDFS-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781388#comment-15781388
]
Hadoop QA commented on HDFS-10759:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-10759 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-10759 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12837786/HDFS-10759.0002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17959/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Change fsimage bool isStriped from boolean to an enum
> -
>
> Key: HDFS-10759
> URL: https://issues.apache.org/jira/browse/HDFS-10759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0-alpha1, 3.0.0-beta1, 3.0.0-alpha2
>Reporter: Ewan Higgs
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10759.0001.patch, HDFS-10759.0002.patch
>
>
> The new erasure coding project has updated the protocol for fsimage such that
> the {{INodeFile}} has a boolean '{{isStriped}}'. I think this is better as an
> enum or integer since a boolean precludes any future block types.
> For example:
> {code}
> enum BlockType {
> CONTIGUOUS = 0,
> STRIPED = 1,
> }
> {code}
> We can also make this more robust to future changes where there are different
> block types supported in a staged rollout. Here, we would use
> {{UNKNOWN_BLOCK_TYPE}} as the first value since this is the default value.
> See
> [here|http://androiddevblog.com/protocol-buffers-pitfall-adding-enum-values/]
> for more discussion.
> {code}
> enum BlockType {
> UNKNOWN_BLOCK_TYPE = 0,
> CONTIGUOUS = 1,
> STRIPED = 2,
> }
> {code}
> But I'm not convinced this is necessary since there are other enums that
> don't use this approach.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11268) Persist erasure coding policy ID in FSImage
[ https://issues.apache.org/jira/browse/HDFS-11268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781380#comment-15781380 ] Wei-Chiu Chuang commented on HDFS-11268: I thought EC policy ID is stored using replication factor field, no? > Persist erasure coding policy ID in FSImage > --- > > Key: HDFS-11268 > URL: https://issues.apache.org/jira/browse/HDFS-11268 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0-alpha1 >Reporter: SammiChen >Assignee: SammiChen > Labels: hdfs-ec-3.0-must-do > > Currently, FSImage only has the information about whether the file is striped > or not. It doesn't save the erasure coding policy ID. Later, when the FSImage > is loaded to create the name space, the default system ec policy is used to > as file's ec policy. In case if the ec policy on file is not the default ec > policy, then the content of the file cannot be accessed correctly in this > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11121) Add assertions to BlockInfo#addStorage to protect from breaking reportedBlock-blockGroup mapping
[
https://issues.apache.org/jira/browse/HDFS-11121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781372#comment-15781372
]
Wei-Chiu Chuang commented on HDFS-11121:
Hi [~tasanuma0829] I don't feel that's necessary as long as we have a test that
covers this case.
> Add assertions to BlockInfo#addStorage to protect from breaking
> reportedBlock-blockGroup mapping
>
>
> Key: HDFS-11121
> URL: https://issues.apache.org/jira/browse/HDFS-11121
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Critical
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-11121.1.patch
>
>
> There are not any assertions in {{BlockInfo.addStorage}}. This may cause that
> {{BlockInfo}} instances accept strange block reports and result in serious
> bugs, like HDFS-10858.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11270) Document the missing options of NameNode bootstrap command
[
https://issues.apache.org/jira/browse/HDFS-11270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781172#comment-15781172
]
Hudson commented on HDFS-11270:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11041 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11041/])
HDFS-11270. Document the missing options of NameNode bootstrap command.
(liuml07: rev c0e0ef29696109af9a018462059f08fd99ee3121)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
> Document the missing options of NameNode bootstrap command
> --
>
> Key: HDFS-11270
> URL: https://issues.apache.org/jira/browse/HDFS-11270
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-11270.001.patch, HDFS-11270.002.patch,
> HDFS-11270.003.patch
>
>
> Currently there are missing three options introduction in {{HDFSCommands.md}}
> of command {{hdfs namenode [-bootstrapStandby]}}. Now this command can
> support {{-force}}, {{-nonInteractive}} and a additional option
> {{-skipSharedEditsCheck}}. It will be better to document what's the meaning
> of these options and that will let users know how to use it correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11270) Document the missing options of NameNode bootstrap command
[
https://issues.apache.org/jira/browse/HDFS-11270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-11270:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.8.0
Status: Resolved (was: Patch Available)
Committed to {{trunk}} through {{branch-2.8}} branches. Thanks for your
contribution [~linyiqun]. Thanks for your helpful review, [~brahmareddy].
> Document the missing options of NameNode bootstrap command
> --
>
> Key: HDFS-11270
> URL: https://issues.apache.org/jira/browse/HDFS-11270
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-11270.001.patch, HDFS-11270.002.patch,
> HDFS-11270.003.patch
>
>
> Currently there are missing three options introduction in {{HDFSCommands.md}}
> of command {{hdfs namenode [-bootstrapStandby]}}. Now this command can
> support {{-force}}, {{-nonInteractive}} and a additional option
> {{-skipSharedEditsCheck}}. It will be better to document what's the meaning
> of these options and that will let users know how to use it correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11271) Typo in NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-11271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781004#comment-15781004 ] Wei-Chiu Chuang commented on HDFS-11271: Thanks, Akira and Brahma! > Typo in NameNode UI > --- > > Key: HDFS-11271 > URL: https://issues.apache.org/jira/browse/HDFS-11271 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, ui >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Trivial > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-11271.001.patch > > > The HADOOP-11098 commit has a typo: > It should be "" instead of "" when max heap is -1. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11275) Check groupEntryIndex and throw a helpful exception on failures when removing ACL.
[
https://issues.apache.org/jira/browse/HDFS-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-11275:
-
Status: Patch Available (was: Open)
> Check groupEntryIndex and throw a helpful exception on failures when removing
> ACL.
> --
>
> Key: HDFS-11275
> URL: https://issues.apache.org/jira/browse/HDFS-11275
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-11275.01.patch
>
>
> We only have the logs to the exception but not the metadata, so hard to
> reproduce. The log is:
> {noformat}
> ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered
> exception on operation
> org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$SetAclOp@7aad3f7d
> java.lang.IndexOutOfBoundsException: index (-4) must not be negative
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:301)
>
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:280)
>
> at
> com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:91)
>
> at
> org.apache.hadoop.hdfs.server.namenode.AclStorage.removeINodeAcl(AclStorage.java:267)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedRemoveAcl(FSDirectory.java:2720)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetAcl(FSDirectory.java:2737)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:827)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:231)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
>
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:846)
> {noformat}
> Let's use this jira to add more information to the exception, to hopefully
> help the next reoccurrence. (Have not failed again in the past week)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11275) Check groupEntryIndex and throw a helpful exception on failures when removing ACL.
[
https://issues.apache.org/jira/browse/HDFS-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-11275:
-
Attachment: HDFS-11275.01.patch
Patch 1 to add the check, don't think unit test is necessary.
Manually checked output by running {{TestFSImageWithAcl}} and changed the check
to always throw: {{Invalid group entry index after binary-searching inode:
/p(16386) with featureEntries:[user:foo:r-x, group::r--]}}
> Check groupEntryIndex and throw a helpful exception on failures when removing
> ACL.
> --
>
> Key: HDFS-11275
> URL: https://issues.apache.org/jira/browse/HDFS-11275
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-11275.01.patch
>
>
> We only have the logs to the exception but not the metadata, so hard to
> reproduce. The log is:
> {noformat}
> ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered
> exception on operation
> org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$SetAclOp@7aad3f7d
> java.lang.IndexOutOfBoundsException: index (-4) must not be negative
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:301)
>
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:280)
>
> at
> com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:91)
>
> at
> org.apache.hadoop.hdfs.server.namenode.AclStorage.removeINodeAcl(AclStorage.java:267)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedRemoveAcl(FSDirectory.java:2720)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetAcl(FSDirectory.java:2737)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:827)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:231)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
>
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:846)
> {noformat}
> Let's use this jira to add more information to the exception, to hopefully
> help the next reoccurrence. (Have not failed again in the past week)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11275) Check groupEntryIndex and throw a helpful exception on failures when removing ACL.
[
https://issues.apache.org/jira/browse/HDFS-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-11275:
-
Description:
We only have the logs to the exception but not the metadata, so hard to
reproduce. The log is:
{noformat}
ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered
exception on operation
org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$SetAclOp@7aad3f7d
java.lang.IndexOutOfBoundsException: index (-4) must not be negative
at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:301)
at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:280)
at
com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:91)
at
org.apache.hadoop.hdfs.server.namenode.AclStorage.removeINodeAcl(AclStorage.java:267)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedRemoveAcl(FSDirectory.java:2720)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetAcl(FSDirectory.java:2737)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:827)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:231)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:846)
{noformat}
Let's use this jira to add more information to the exception, to hopefully help
the next reoccurrence. (Have not failed again in the past week)
> Check groupEntryIndex and throw a helpful exception on failures when removing
> ACL.
> --
>
> Key: HDFS-11275
> URL: https://issues.apache.org/jira/browse/HDFS-11275
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>
> We only have the logs to the exception but not the metadata, so hard to
> reproduce. The log is:
> {noformat}
> ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered
> exception on operation
> org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$SetAclOp@7aad3f7d
> java.lang.IndexOutOfBoundsException: index (-4) must not be negative
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:301)
>
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:280)
>
> at
> com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:91)
>
> at
> org.apache.hadoop.hdfs.server.namenode.AclStorage.removeINodeAcl(AclStorage.java:267)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedRemoveAcl(FSDirectory.java:2720)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetAcl(FSDirectory.java:2737)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:827)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:231)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
>
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:846)
> {noformat}
> Let's use this jira to add more information to the exception, to hopefully
> help the next reoccurrence. (Have not failed again in the past week)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11275) Check groupEntryIndex and throw a helpful exception on failures when removing ACL.
Xiao Chen created HDFS-11275: Summary: Check groupEntryIndex and throw a helpful exception on failures when removing ACL. Key: HDFS-11275 URL: https://issues.apache.org/jira/browse/HDFS-11275 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Xiao Chen Assignee: Xiao Chen -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11150) [SPS]: Provide persistence when satisfying storage policy.
[ https://issues.apache.org/jira/browse/HDFS-11150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15780118#comment-15780118 ] Yuanbo Liu commented on HDFS-11150: --- come back from chrismas holiday. Will upload a new patch this week. Sorry for the late response. > [SPS]: Provide persistence when satisfying storage policy. > -- > > Key: HDFS-11150 > URL: https://issues.apache.org/jira/browse/HDFS-11150 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu > Attachments: HDFS-11150-HDFS-10285.001.patch, > HDFS-11150-HDFS-10285.002.patch, editsStored, editsStored.xml > > > Provide persistence for SPS in case that Hadoop cluster crashes by accident. > Basically we need to change EditLog and FsImage here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
