[jira] [Commented] (HDFS-12389) Ozone: oz commandline list calls should return valid JSON format output
[
https://issues.apache.org/jira/browse/HDFS-12389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162413#comment-16162413
]
Weiwei Yang commented on HDFS-12389:
Hi [~anu], [~linyiqun], [~vagarychen]
Excuse me for interrupting, could you please help to review this patch when you
have time? This is pretty straightforward but would be helpful to folks who
wants to play with CLI outputs. Thanks a lot.
> Ozone: oz commandline list calls should return valid JSON format output
> ---
>
> Key: HDFS-12389
> URL: https://issues.apache.org/jira/browse/HDFS-12389
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Labels: ozoneMerge
> Attachments: HDFS-12389-HDFS-7240.001.patch,
> HDFS-12389-HDFS-7240.002.patch, json_output_test.log
>
>
> At present the outputs of {{listVolume}}, {{listBucket}} and {{listKey}} are
> hard to parse, for example following call
> {code}
> ./bin/hdfs oz -listVolume http://localhost:9864 -user wwei
> {code}
> lists all volumes in my cluster and it returns
> {noformat}
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "modifiedOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "size" : 10240,
> "keyName" : "key-0-22381",
> "dataFileName" : null
> }
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "modifiedOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "size" : 10240,
> "keyName" : "key-0-22381",
> "dataFileName" : null
> }
> ...
> {noformat}
> this is not a valid JSON format output hence it is hard to parse in clients'
> script for further interactions. Propose to reformat them to valid JSON data.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12017) Ozone: Container: Move IPC port to 98xx range
[ https://issues.apache.org/jira/browse/HDFS-12017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-12017: Priority: Major (was: Trivial) > Ozone: Container: Move IPC port to 98xx range > - > > Key: HDFS-12017 > URL: https://issues.apache.org/jira/browse/HDFS-12017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: ozoneMerge > > We use port 50011 port -- This is an old format of choosing port. Hadoop 3 > has already moved to port ranges under 98xx. In fact all of KSM/SCM and > CBlock is using 98xx range. We should move 50011 to a port under 98xx. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12412) Remove ErasureCodingWorker.stripedReadPool
[
https://issues.apache.org/jira/browse/HDFS-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-12412:
-
Attachment: HDFS-12412.00.patch
Change {{stripedReadPool}} to an unbound {{cacheThreadPool}}. [~andrew.wang],
[~drankye] could you give a review?
Thanks.
> Remove ErasureCodingWorker.stripedReadPool
> --
>
> Key: HDFS-12412
> URL: https://issues.apache.org/jira/browse/HDFS-12412
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-12412.00.patch
>
>
> In {{ErasureCodingWorker}}, it uses {{stripedReconstructionPool}} to schedule
> the EC recovery tasks, while uses {{stripedReadPool}} for the reader threads
> in each recovery task. We only need one of them to throttle the speed of
> recovery process, because each EC recovery task has a fix number of source
> readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044, the
> speed of EC recovery can be throttled by {{strippedReconstructionPool}} with
> {{xmitsInProgress}}.
> Moreover, keeping {{stripedReadPool}} makes customer difficult to understand
> and calculate the right balance between
> {{dfs.datanode.ec.reconstruction.stripedread.threads}},
> {{dfs.datanode.ec.reconstruction.stripedblock.threads.size}} and
> {{maxReplicationStreams}}. For example, a small {{stripread.threads}}
> (comparing to which {{reconstruction.threads.size}} implies), will
> unnecessarily limit the speed of recovery, which leads to larger MTTR.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11676) Ozone: SCM CLI: Implement close container command
[
https://issues.apache.org/jira/browse/HDFS-11676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162167#comment-16162167
]
Hadoop QA commented on HDFS-11676:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HDFS-11676 does not apply to HDFS-7240. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-11676 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886517/HDFS-11676-HDFS-7240.002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/21080/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: SCM CLI: Implement close container command
> -
>
> Key: HDFS-11676
> URL: https://issues.apache.org/jira/browse/HDFS-11676
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Chen Liang
> Labels: OzonePostMerge, tocheck
> Attachments: HDFS-11676-HDFS-7240.001.patch,
> HDFS-11676-HDFS-7240.002.patch
>
>
> Implement close container command
> {code}
> hdfs scm -container close
> {code}
> This command connects to SCM and closes a container. Once the container is
> closed in the SCM, the corresponding container is closed at the appropriate
> datanode. if the container does not exist, it will return an error.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11676) Ozone: SCM CLI: Implement close container command
[
https://issues.apache.org/jira/browse/HDFS-11676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-11676:
--
Attachment: HDFS-11676-HDFS-7240.003.patch
v003 patch to rebase.
> Ozone: SCM CLI: Implement close container command
> -
>
> Key: HDFS-11676
> URL: https://issues.apache.org/jira/browse/HDFS-11676
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Chen Liang
> Labels: OzonePostMerge, tocheck
> Attachments: HDFS-11676-HDFS-7240.001.patch,
> HDFS-11676-HDFS-7240.002.patch, HDFS-11676-HDFS-7240.003.patch
>
>
> Implement close container command
> {code}
> hdfs scm -container close
> {code}
> This command connects to SCM and closes a container. Once the container is
> closed in the SCM, the corresponding container is closed at the appropriate
> datanode. if the container does not exist, it will return an error.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11851) getGlobalJNIEnv() may deadlock if exception is thrown
[
https://issues.apache.org/jira/browse/HDFS-11851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162252#comment-16162252
]
John Zhuge commented on HDFS-11851:
---
This fixes a deadlock introduced by HDFS-11529 which is only in trunk, not in
2.6.
> getGlobalJNIEnv() may deadlock if exception is thrown
> -
>
> Key: HDFS-11851
> URL: https://issues.apache.org/jira/browse/HDFS-11851
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 3.0.0-alpha4
>Reporter: Henry Robinson
>Assignee: Sailesh Mukil
>Priority: Blocker
> Fix For: 3.0.0-alpha4
>
> Attachments: HDFS-11851.000.patch, HDFS-11851.001.patch,
> HDFS-11851.002.patch, HDFS-11851.003.patch, HDFS-11851.004.patch,
> HDFS-11851.005.patch
>
>
> HDFS-11529 introduced a deadlock into {{getGlobalJNIEnv()}} if an exception
> is thrown. {{getGlobalJNIEnv()}} holds {{jvmMutex}}, but
> {{printExceptionAndFree()}} will eventually try to acquire that lock in
> {{setTLSExceptionStrings()}}.
> The exception might get caught from {{loadFileSystems}}:
> {code}
> jthr = invokeMethod(env, NULL, STATIC, NULL,
> "org/apache/hadoop/fs/FileSystem",
> "loadFileSystems", "()V");
> if (jthr) {
> printExceptionAndFree(env, jthr, PRINT_EXC_ALL,
> "loadFileSystems");
> }
> }
> {code}
> and here's the relevant parts of the stack trace from where I call this API
> in Impala, which uses {{libhdfs}}:
> {code}
> #0 __lll_lock_wait () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
> #1 0x74a8d657 in _L_lock_909 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #2 0x74a8d480 in __GI___pthread_mutex_lock (mutex=0x47ce960
> ) at ../nptl/pthread_mutex_lock.c:79
> #3 0x02f06056 in mutexLock (m=) at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/mutexes.c:28
> #4 0x02efe817 in setTLSExceptionStrings (rootCause=0x0,
> stackTrace=0x0) at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:581
> #5 0x02f065d7 in printExceptionAndFreeV (env=0x513c1e8,
> exc=0x508a8c0, noPrintFlags=, fmt=0x34349cf "loadFileSystems",
> ap=0x7fffb660)
> at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:183
> #6 0x02f0683d in printExceptionAndFree (env=,
> exc=, noPrintFlags=, fmt=)
> at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:213
> #7 0x02eff60f in getGlobalJNIEnv () at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:463
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12407) Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or JournalNodeRpcServer fails to start
[
https://issues.apache.org/jira/browse/HDFS-12407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162324#comment-16162324
]
Hadoop QA commented on HDFS-12407:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 17 unchanged - 1 fixed = 18 total (was 18) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}159m 14s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}187m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 |
| | hadoop.hdfs.TestFileAppendRestart |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| |
[jira] [Commented] (HDFS-12323) NameNode terminates after full GC thinking QJM unresponsive if full GC is much longer than timeout
[
https://issues.apache.org/jira/browse/HDFS-12323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162384#comment-16162384
]
Konstantin Shvachko commented on HDFS-12323:
The change looks good you are increasing timeout by the actual GC pause.
My only concerns that you removed private no-argument constructor
{{QuorumCall()}}, which can lead to different problems, like if it is used
somewhere via reflections or if you subclass {{QuorumCall}} in the future. May
be better to add it back saying explicitly not to use anywhere.
> NameNode terminates after full GC thinking QJM unresponsive if full GC is
> much longer than timeout
> --
>
> Key: HDFS-12323
> URL: https://issues.apache.org/jira/browse/HDFS-12323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, qjm
>Affects Versions: 2.7.4
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: HDFS-12323.000.patch, HDFS-12323.001.patch
>
>
> HDFS-10733 attempted to fix the issue where the Namenode process would
> terminate itself if it had a GC pause which lasted longer than the QJM
> timeout, since it would think that the QJM had taken too long to respond.
> However, it only bumps up the timeout expiration by one timeout length, so if
> the GC pause was e.g. 2x the length of the timeout, a TimeoutException will
> be thrown and the NN will still terminate itself.
> Thanks to [~yangjiandan] for noting this issue as a comment on HDFS-10733; we
> have also seen this issue on a real cluster even after HDFS-10733 is applied.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11772) Ozone: KSM: set creationTime for volume/bucket/key
[ https://issues.apache.org/jira/browse/HDFS-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162405#comment-16162405 ] Anu Engineer commented on HDFS-11772: - [~linyiqun] Thank you. I will resolve this issue now. > Ozone: KSM: set creationTime for volume/bucket/key > -- > > Key: HDFS-11772 > URL: https://issues.apache.org/jira/browse/HDFS-11772 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh > Labels: ozoneMerge > > Returns the volume information from KSM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12389) Ozone: oz commandline list calls should return valid JSON format output
[
https://issues.apache.org/jira/browse/HDFS-12389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162422#comment-16162422
]
Anu Engineer commented on HDFS-12389:
-
+1, Thanks for fixing this. That was a dumb mistake. Really appreciate you
fixing it. Thanks for flagging to get the code review attention too.
> Ozone: oz commandline list calls should return valid JSON format output
> ---
>
> Key: HDFS-12389
> URL: https://issues.apache.org/jira/browse/HDFS-12389
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Labels: ozoneMerge
> Attachments: HDFS-12389-HDFS-7240.001.patch,
> HDFS-12389-HDFS-7240.002.patch, json_output_test.log
>
>
> At present the outputs of {{listVolume}}, {{listBucket}} and {{listKey}} are
> hard to parse, for example following call
> {code}
> ./bin/hdfs oz -listVolume http://localhost:9864 -user wwei
> {code}
> lists all volumes in my cluster and it returns
> {noformat}
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "modifiedOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "size" : 10240,
> "keyName" : "key-0-22381",
> "dataFileName" : null
> }
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "modifiedOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "size" : 10240,
> "keyName" : "key-0-22381",
> "dataFileName" : null
> }
> ...
> {noformat}
> this is not a valid JSON format output hence it is hard to parse in clients'
> script for further interactions. Propose to reformat them to valid JSON data.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12389) Ozone: oz commandline list calls should return valid JSON format output
[
https://issues.apache.org/jira/browse/HDFS-12389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162435#comment-16162435
]
Weiwei Yang commented on HDFS-12389:
Thanks [~anu] for the quick review, I have committed this to the feature branch.
> Ozone: oz commandline list calls should return valid JSON format output
> ---
>
> Key: HDFS-12389
> URL: https://issues.apache.org/jira/browse/HDFS-12389
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Labels: ozoneMerge
> Fix For: HDFS-7240
>
> Attachments: HDFS-12389-HDFS-7240.001.patch,
> HDFS-12389-HDFS-7240.002.patch, json_output_test.log
>
>
> At present the outputs of {{listVolume}}, {{listBucket}} and {{listKey}} are
> hard to parse, for example following call
> {code}
> ./bin/hdfs oz -listVolume http://localhost:9864 -user wwei
> {code}
> lists all volumes in my cluster and it returns
> {noformat}
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "modifiedOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "size" : 10240,
> "keyName" : "key-0-22381",
> "dataFileName" : null
> }
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "modifiedOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "size" : 10240,
> "keyName" : "key-0-22381",
> "dataFileName" : null
> }
> ...
> {noformat}
> this is not a valid JSON format output hence it is hard to parse in clients'
> script for further interactions. Propose to reformat them to valid JSON data.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12414) Ensure to use CLI command to enable/disable erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-12414: - Status: Open (was: Patch Available) > Ensure to use CLI command to enable/disable erasure coding policy > - > > Key: HDFS-12414 > URL: https://issues.apache.org/jira/browse/HDFS-12414 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: SammiChen > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12414.001.patch, HDFS-12414.002.patch > > > Currently, there are two methods for user to enable/disable a erasure coding > policy. One is through "dfs.namenode.ec.policies.enabled" property which is a > static way to configure the enabled erasure coding policies. Another is > through "enableErasureCodingPolicy" or "disabledErasureCodingPolicy" API > which can enabled or disable erasure coding policy at runtime. > When Namenode restart, there is potential state conflicts between the policy > defined in "dfs.namenode.ec.policies.enabled" and policy saved in fsImage. To > resolve the conflict and simplify the operation, it's better to use just one > way and remove the old method configuring the property. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12417) Disable flaky TestDFSStripedOutputStreamWithFailure
[ https://issues.apache.org/jira/browse/HDFS-12417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HDFS-12417: -- Assignee: Andrew Wang > Disable flaky TestDFSStripedOutputStreamWithFailure > --- > > Key: HDFS-12417 > URL: https://issues.apache.org/jira/browse/HDFS-12417 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Chris Douglas >Assignee: Andrew Wang > Attachments: HDFS-12417.000.patch, HDFS-12417.001.patch, > HDFS-12417.002.patch > > > Some subset of TestDFSStripedOutputStreamWithFailure tests almost always fail > in test-patch runs. Since its failure is no longer seen as a blocker for > commit, it should be disabled until it is more reliable. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12412) Remove ErasureCodingWorker.stripedReadPool
[
https://issues.apache.org/jira/browse/HDFS-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162204#comment-16162204
]
Andrew Wang commented on HDFS-12412:
Thanks for working on this Eddy. I did a grep for the removed config key:
{noformat}
-> % ag "stripedread.threads"
hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
140: 1. `dfs.datanode.ec.reconstruction.stripedread.threads` - Number of
concurrent reader threads. Default value is 20 threads.
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
3052: dfs.datanode.ec.reconstruction.stripedread.threads
{noformat}
We need to update these as well. Otherwise, +1 LGTM!
> Remove ErasureCodingWorker.stripedReadPool
> --
>
> Key: HDFS-12412
> URL: https://issues.apache.org/jira/browse/HDFS-12412
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-12412.00.patch
>
>
> In {{ErasureCodingWorker}}, it uses {{stripedReconstructionPool}} to schedule
> the EC recovery tasks, while uses {{stripedReadPool}} for the reader threads
> in each recovery task. We only need one of them to throttle the speed of
> recovery process, because each EC recovery task has a fix number of source
> readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044, the
> speed of EC recovery can be throttled by {{strippedReconstructionPool}} with
> {{xmitsInProgress}}.
> Moreover, keeping {{stripedReadPool}} makes customer difficult to understand
> and calculate the right balance between
> {{dfs.datanode.ec.reconstruction.stripedread.threads}},
> {{dfs.datanode.ec.reconstruction.stripedblock.threads.size}} and
> {{maxReplicationStreams}}. For example, a small {{stripread.threads}}
> (comparing to which {{reconstruction.threads.size}} implies), will
> unnecessarily limit the speed of recovery, which leads to larger MTTR.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12409) Add metrics of execution time of different stages in EC recovery task
[ https://issues.apache.org/jira/browse/HDFS-12409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-12409: - Attachment: HDFS-12409.01.patch Fix checkstyle warnings. The findbug warnings are not relevant and will fix in another JIRA. > Add metrics of execution time of different stages in EC recovery task > - > > Key: HDFS-12409 > URL: https://issues.apache.org/jira/browse/HDFS-12409 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu >Priority: Minor > Attachments: HDFS-12409.00.patch, HDFS-12409.01.patch > > > Admin can use more metrics to monitor EC recovery tasks, to get insights to > tune recovery performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11851) getGlobalJNIEnv() may deadlock if exception is thrown
[
https://issues.apache.org/jira/browse/HDFS-11851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162260#comment-16162260
]
Ruslan Dautkhanov commented on HDFS-11851:
--
[~jzhuge], I have following exception that matches HDFS-11851, doesn't it? See
below:
{code}
$ gdb -p 8248
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7
/cut/
(gdb)
(gdb) bt
#0 0x7fddd9f141bd in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x7fddd9f0fd02 in _L_lock_791 () from /lib64/libpthread.so.0
#2 0x7fddd9f0fc08 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x7fddda9f3e26 in mutexLock () from
/opt/cloudera/parcels/CDH/lib64/libhdfs.so.0.0.0
#4 0x7fddda9ed6f1 in setTLSExceptionStrings () from
/opt/cloudera/parcels/CDH/lib64/libhdfs.so.0.0.0
#5 0x7fddda9ec38c in printExceptionAndFreeV () from
/opt/cloudera/parcels/CDH/lib64/libhdfs.so.0.0.0
#6 0x7fddda9ec52d in printExceptionAndFree () from
/opt/cloudera/parcels/CDH/lib64/libhdfs.so.0.0.0
#7 0x7fddda9ed46b in getJNIEnv () from
/opt/cloudera/parcels/CDH/lib64/libhdfs.so.0.0.0
#8 0x7fddda9eee94 in hdfsBuilderConnect () from
/opt/cloudera/parcels/CDH/lib64/libhdfs.so.0.0.0
#9 0x00400950 in main ()
{code}
It is from a Hadoop-2.6 based distribution of Hadoop (CDH 5.10).
> getGlobalJNIEnv() may deadlock if exception is thrown
> -
>
> Key: HDFS-11851
> URL: https://issues.apache.org/jira/browse/HDFS-11851
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 3.0.0-alpha4
>Reporter: Henry Robinson
>Assignee: Sailesh Mukil
>Priority: Blocker
> Fix For: 3.0.0-alpha4
>
> Attachments: HDFS-11851.000.patch, HDFS-11851.001.patch,
> HDFS-11851.002.patch, HDFS-11851.003.patch, HDFS-11851.004.patch,
> HDFS-11851.005.patch
>
>
> HDFS-11529 introduced a deadlock into {{getGlobalJNIEnv()}} if an exception
> is thrown. {{getGlobalJNIEnv()}} holds {{jvmMutex}}, but
> {{printExceptionAndFree()}} will eventually try to acquire that lock in
> {{setTLSExceptionStrings()}}.
> The exception might get caught from {{loadFileSystems}}:
> {code}
> jthr = invokeMethod(env, NULL, STATIC, NULL,
> "org/apache/hadoop/fs/FileSystem",
> "loadFileSystems", "()V");
> if (jthr) {
> printExceptionAndFree(env, jthr, PRINT_EXC_ALL,
> "loadFileSystems");
> }
> }
> {code}
> and here's the relevant parts of the stack trace from where I call this API
> in Impala, which uses {{libhdfs}}:
> {code}
> #0 __lll_lock_wait () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
> #1 0x74a8d657 in _L_lock_909 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #2 0x74a8d480 in __GI___pthread_mutex_lock (mutex=0x47ce960
> ) at ../nptl/pthread_mutex_lock.c:79
> #3 0x02f06056 in mutexLock (m=) at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/mutexes.c:28
> #4 0x02efe817 in setTLSExceptionStrings (rootCause=0x0,
> stackTrace=0x0) at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:581
> #5 0x02f065d7 in printExceptionAndFreeV (env=0x513c1e8,
> exc=0x508a8c0, noPrintFlags=, fmt=0x34349cf "loadFileSystems",
> ap=0x7fffb660)
> at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:183
> #6 0x02f0683d in printExceptionAndFree (env=,
> exc=, noPrintFlags=, fmt=)
> at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:213
> #7 0x02eff60f in getGlobalJNIEnv () at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:463
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12389) Ozone: oz commandline list calls should return valid JSON format output
[
https://issues.apache.org/jira/browse/HDFS-12389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12389:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-7240
Status: Resolved (was: Patch Available)
> Ozone: oz commandline list calls should return valid JSON format output
> ---
>
> Key: HDFS-12389
> URL: https://issues.apache.org/jira/browse/HDFS-12389
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Labels: ozoneMerge
> Fix For: HDFS-7240
>
> Attachments: HDFS-12389-HDFS-7240.001.patch,
> HDFS-12389-HDFS-7240.002.patch, json_output_test.log
>
>
> At present the outputs of {{listVolume}}, {{listBucket}} and {{listKey}} are
> hard to parse, for example following call
> {code}
> ./bin/hdfs oz -listVolume http://localhost:9864 -user wwei
> {code}
> lists all volumes in my cluster and it returns
> {noformat}
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "modifiedOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "size" : 10240,
> "keyName" : "key-0-22381",
> "dataFileName" : null
> }
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "modifiedOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "size" : 10240,
> "keyName" : "key-0-22381",
> "dataFileName" : null
> }
> ...
> {noformat}
> this is not a valid JSON format output hence it is hard to parse in clients'
> script for further interactions. Propose to reformat them to valid JSON data.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-12417) Disable flaky TestDFSStripedOutputStreamWithFailure
[ https://issues.apache.org/jira/browse/HDFS-12417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-12417: -- Comment: was deleted (was: Hi Andrew, There is a problem with this patch, it is causing a failure due to maven enforce plugin rules. ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (enforce-banned-dependencies) on project hadoop-client-check-test-invariants: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1] [WARNING] Rule 0: org.apache.maven.plugins.enforcer.BanTransitiveDependencies failed with message: org.apache.hadoop:hadoop-client-check-test-invariants:pom:3.1.0-SNAPSHOT org.apache.hadoop:hadoop-client-minicluster:jar:3.1.0-SNAPSHOT:compile has transitive dependencies: junit:junit:jar:4.11:runtime [excluded] org.mockito:mockito-all:jar:1.8.5:compile ) > Disable flaky TestDFSStripedOutputStreamWithFailure > --- > > Key: HDFS-12417 > URL: https://issues.apache.org/jira/browse/HDFS-12417 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Chris Douglas >Assignee: Andrew Wang > Attachments: HDFS-12417.000.patch, HDFS-12417.001.patch, > HDFS-12417.002.patch > > > Some subset of TestDFSStripedOutputStreamWithFailure tests almost always fail > in test-patch runs. Since its failure is no longer seen as a blocker for > commit, it should be disabled until it is more reliable. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12417) Disable flaky TestDFSStripedOutputStreamWithFailure
[ https://issues.apache.org/jira/browse/HDFS-12417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162142#comment-16162142 ] Bharat Viswanadham commented on HDFS-12417: --- Hi Andrew, There is a problem with this patch, it is causing a failure due to maven enforce plugin rules. ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (enforce-banned-dependencies) on project hadoop-client-check-test-invariants: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1] [WARNING] Rule 0: org.apache.maven.plugins.enforcer.BanTransitiveDependencies failed with message: org.apache.hadoop:hadoop-client-check-test-invariants:pom:3.1.0-SNAPSHOT org.apache.hadoop:hadoop-client-minicluster:jar:3.1.0-SNAPSHOT:compile has transitive dependencies: junit:junit:jar:4.11:runtime [excluded] org.mockito:mockito-all:jar:1.8.5:compile > Disable flaky TestDFSStripedOutputStreamWithFailure > --- > > Key: HDFS-12417 > URL: https://issues.apache.org/jira/browse/HDFS-12417 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Chris Douglas >Assignee: Andrew Wang > Attachments: HDFS-12417.000.patch, HDFS-12417.001.patch, > HDFS-12417.002.patch > > > Some subset of TestDFSStripedOutputStreamWithFailure tests almost always fail > in test-patch runs. Since its failure is no longer seen as a blocker for > commit, it should be disabled until it is more reliable. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11676) Ozone: SCM CLI: Implement close container command
[
https://issues.apache.org/jira/browse/HDFS-11676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-11676:
--
Attachment: (was: HDFS-11676-HDFS-7240.002.patch)
> Ozone: SCM CLI: Implement close container command
> -
>
> Key: HDFS-11676
> URL: https://issues.apache.org/jira/browse/HDFS-11676
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Chen Liang
> Labels: OzonePostMerge, tocheck
> Attachments: HDFS-11676-HDFS-7240.001.patch
>
>
> Implement close container command
> {code}
> hdfs scm -container close
> {code}
> This command connects to SCM and closes a container. Once the container is
> closed in the SCM, the corresponding container is closed at the appropriate
> datanode. if the container does not exist, it will return an error.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11600) Refactor TestDFSStripedOutputStreamWithFailure test classes
[
https://issues.apache.org/jira/browse/HDFS-11600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162192#comment-16162192
]
Chris Douglas commented on HDFS-11600:
--
bq. I posted an alternative disable patch on HDFS-12417
Sure. No reason to block CI stability on a test refactor. Cancelled patch.
> Refactor TestDFSStripedOutputStreamWithFailure test classes
> ---
>
> Key: HDFS-11600
> URL: https://issues.apache.org/jira/browse/HDFS-11600
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Affects Versions: 3.0.0-alpha2
>Reporter: Andrew Wang
>Assignee: Elek, Marton
>Priority: Minor
> Attachments: HDFS-11600.002.patch, HDFS-11600-1.patch
>
>
> TestDFSStripedOutputStreamWithFailure has a great number of subclasses. The
> tests are parameterized based on the name of these subclasses.
> Seems like we could parameterize these tests with JUnit and then not need all
> these separate test classes.
> Another note, the tests will randomly return instead of running the test.
> Using {{Assume}} instead would make it more clear in the test output that
> these tests were skipped.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12420) Disable Namenode format when data already exist
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HDFS-12420:
--
Attachment: HDFS-12420.01.patch
> Disable Namenode format when data already exist
> ---
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12420.01.patch
>
>
> Disable NameNode format to avoid accidental formatting of Namenode in
> production cluster. If someone really wants to delete the complete fsImage
> they can first delete the metadata dir and then run {code} hdfs namenode
> -format{code} manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-12420) Disable Namenode format when data already exist
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-12420 started by Ajay Kumar.
-
> Disable Namenode format when data already exist
> ---
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12420.01.patch
>
>
> Disable NameNode format to avoid accidental formatting of Namenode in
> production cluster. If someone really wants to delete the complete fsImage
> they can first delete the metadata dir and then run {code} hdfs namenode
> -format{code} manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work stopped] (HDFS-12420) Disable Namenode format when data already exist
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-12420 stopped by Ajay Kumar.
-
> Disable Namenode format when data already exist
> ---
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12420.01.patch
>
>
> Disable NameNode format to avoid accidental formatting of Namenode in
> production cluster. If someone really wants to delete the complete fsImage
> they can first delete the metadata dir and then run {code} hdfs namenode
> -format{code} manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12409) Add metrics of execution time of different stages in EC recovery task
[
https://issues.apache.org/jira/browse/HDFS-12409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162298#comment-16162298
]
Hadoop QA commented on HDFS-12409:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 3 new + 68 unchanged - 0 fixed = 71 total (was 68) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}137m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.TestEncryptionZonesWithHA |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
| | hadoop.hdfs.server.namenode.TestReencryption |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| |
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
|
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12409 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886170/HDFS-12409.00.patch |
| Optional Tests |
[jira] [Assigned] (HDFS-11613) Ozone: Cleanup findbugs issues
[ https://issues.apache.org/jira/browse/HDFS-11613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey reassigned HDFS-11613: --- Assignee: Lokesh Jain > Ozone: Cleanup findbugs issues > -- > > Key: HDFS-11613 > URL: https://issues.apache.org/jira/browse/HDFS-11613 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Lokesh Jain >Priority: Blocker > Labels: ozoneMerge > > Some of the ozone checkins happened before Findbugs started running on test > files. This will cause issues when we attempt to merge with trunk. This jira > tracks cleaning up all Findbugs issue under ozone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11612) Ozone: Cleanup Checkstyle issues
[ https://issues.apache.org/jira/browse/HDFS-11612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey reassigned HDFS-11612: --- Assignee: Shashikant Banerjee > Ozone: Cleanup Checkstyle issues > > > Key: HDFS-11612 > URL: https://issues.apache.org/jira/browse/HDFS-11612 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Shashikant Banerjee >Priority: Blocker > Labels: ozoneMerge > > There is a bunch of check style issues under Ozone tree. We have to clean > them up before we call for a merge of this tree. This jira tracks that work > item. It would be a noisy but mostly content less change. Hence it is easier > to track that in separate patch -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11676) Ozone: SCM CLI: Implement close container command
[
https://issues.apache.org/jira/browse/HDFS-11676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162313#comment-16162313
]
Hadoop QA commented on HDFS-11676:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
32s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
27s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
24s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 4s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestFileAppendRestart |
| | hadoop.hdfs.TestWriteReadStripedFile |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-11676 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886521/HDFS-11676-HDFS-7240.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 7411ebe3cf67 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDFS-11676) Ozone: SCM CLI: Implement close container command
[
https://issues.apache.org/jira/browse/HDFS-11676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162365#comment-16162365
]
Weiwei Yang commented on HDFS-11676:
+1 on the v3 patch, thanks [~vagarychen] for the patch and [~linyiqun] for the
review. If there is no further comments, I will commit this few hours later,
thanks all.
> Ozone: SCM CLI: Implement close container command
> -
>
> Key: HDFS-11676
> URL: https://issues.apache.org/jira/browse/HDFS-11676
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Chen Liang
> Labels: OzonePostMerge, tocheck
> Attachments: HDFS-11676-HDFS-7240.001.patch,
> HDFS-11676-HDFS-7240.002.patch, HDFS-11676-HDFS-7240.003.patch
>
>
> Implement close container command
> {code}
> hdfs scm -container close
> {code}
> This command connects to SCM and closes a container. Once the container is
> closed in the SCM, the corresponding container is closed at the appropriate
> datanode. if the container does not exist, it will return an error.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11676) Ozone: SCM CLI: Implement close container command
[
https://issues.apache.org/jira/browse/HDFS-11676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162386#comment-16162386
]
Yiqun Lin commented on HDFS-11676:
--
Also +1 from me.
> Ozone: SCM CLI: Implement close container command
> -
>
> Key: HDFS-11676
> URL: https://issues.apache.org/jira/browse/HDFS-11676
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Chen Liang
> Labels: OzonePostMerge, tocheck
> Attachments: HDFS-11676-HDFS-7240.001.patch,
> HDFS-11676-HDFS-7240.002.patch, HDFS-11676-HDFS-7240.003.patch
>
>
> Implement close container command
> {code}
> hdfs scm -container close
> {code}
> This command connects to SCM and closes a container. Once the container is
> closed in the SCM, the corresponding container is closed at the appropriate
> datanode. if the container does not exist, it will return an error.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12328) Ozone: Purge metadata of deleted blocks after max retry times
[
https://issues.apache.org/jira/browse/HDFS-12328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162393#comment-16162393
]
Weiwei Yang commented on HDFS-12328:
Hi [~yuanbo]
Thanks for updating the description, I noticed that you proposed to introduce a
new sub command for scm "-txid", well I am not in favor of this. The reason is
the TXs are internal notions, we don't need to expose this to end user. When a
block cannot be deleted after max time of retries, we consider this block is
*corrupted*, from user level, I think we need a *block* level command in SCM.
Some initial thoughts
{code}
// list all corrupted block IDs
hdfs scm -block -list --corrupted
// get detail info of this block as much as possible, where the data locates
// so admin can logon to certain datanode to debug why deletion was failed
hdfs scm -block -info xxx
// delete a certain block
hdfs scm -block -delete xxx
// delete all corrupted blocks
// this will need extra confirmation from keyboard by user
hdfs scm -block -delete --corrupted
{code}
I have set the priority to major, because I don't think this is a super
important feature that must be addressed now (lets get this done as a post
merge task). At present, we have alternative to leverage SQLCli to dump DB info
to debug. Also like [~linyiqun] commented, it might be good to start with
adding corrupted blocks in SCM JMX which is a smaller task and that can help us
understand how big the problem is here.
Thanks
> Ozone: Purge metadata of deleted blocks after max retry times
> -
>
> Key: HDFS-12328
> URL: https://issues.apache.org/jira/browse/HDFS-12328
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yuanbo Liu
>Assignee: Yuanbo Liu
> Labels: OzonePostMerge
>
> In HDFS-12283, we set the value of count to -1 if blocks cannot be deleted
> after max retry times. We need to provide APIs for admins to purge the "-1"
> metadata manually. Implement these commands:
> list the txids
> {code}
> hdfs scm -txid list -count -retry
> {code}
> delete the txid
> {code}
> hdfs scm -txid delete -id
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12423) Ozone: TopN container choosing policy should ignore unnecessary containers
[
https://issues.apache.org/jira/browse/HDFS-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-12423:
-
Description:
TopN container choosing policy should ignore unnecessary containers. Currently
TopN policy selects specified count of containers but not check the number of
pending deletion blocks. So there is a chance there will be some unnecessary
containers being chosen. That is say we will choose some containers that don't
include any pending deletion blocks. The related output:
{noformat}
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion, pending
deletion blocks num: 1.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion, pending
deletion blocks num: 0.
{noformat}
We should ignore these containers which pending deletion blocks num is 0, this
can reduce some unnecessary block-deletion iterations.
was:
TopN container choosing policy should ignore unnecessary containers. Currently
TopN policy selects specified count of containers but not check the number of
pending deletion blocks. So there is a chance there will be some unnecessary
containers being chosen. That is say we will choose some containers that don't
include any pending deletion blocks. The related output:
{noformat}
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion, pending
deletion blocks num: 1.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion, pending
deletion blocks num: 0.
{noformat}
We should ignore these containers which pending deletion blocks num is 0, this
can reduce some unnecessary iterations.
> Ozone: TopN container choosing policy should ignore unnecessary containers
> --
>
> Key: HDFS-12423
> URL: https://issues.apache.org/jira/browse/HDFS-12423
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>
> TopN container choosing policy should ignore unnecessary containers.
> Currently TopN policy selects specified count of containers but not check the
> number of pending deletion blocks. So there is a chance there will be some
> unnecessary containers being chosen. That is say we will choose some
> containers that don't include any pending deletion blocks. The related output:
> {noformat}
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion,
> pending deletion blocks num: 1.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion,
> pending deletion blocks num: 0.
> {noformat}
> We should ignore these containers which pending deletion blocks num is 0,
> this can reduce some unnecessary block-deletion
[jira] [Updated] (HDFS-11676) Ozone: SCM CLI: Implement close container command
[
https://issues.apache.org/jira/browse/HDFS-11676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-11676:
--
Attachment: HDFS-11676-HDFS-7240.002.patch
> Ozone: SCM CLI: Implement close container command
> -
>
> Key: HDFS-11676
> URL: https://issues.apache.org/jira/browse/HDFS-11676
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Chen Liang
> Labels: OzonePostMerge, tocheck
> Attachments: HDFS-11676-HDFS-7240.001.patch,
> HDFS-11676-HDFS-7240.002.patch
>
>
> Implement close container command
> {code}
> hdfs scm -container close
> {code}
> This command connects to SCM and closes a container. Once the container is
> closed in the SCM, the corresponding container is closed at the appropriate
> datanode. if the container does not exist, it will return an error.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11851) getGlobalJNIEnv() may deadlock if exception is thrown
[
https://issues.apache.org/jira/browse/HDFS-11851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162251#comment-16162251
]
Ruslan Dautkhanov commented on HDFS-11851:
--
Would it be possible to backport this patch to HDFS 2.6 ? Thanks.
> getGlobalJNIEnv() may deadlock if exception is thrown
> -
>
> Key: HDFS-11851
> URL: https://issues.apache.org/jira/browse/HDFS-11851
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 3.0.0-alpha4
>Reporter: Henry Robinson
>Assignee: Sailesh Mukil
>Priority: Blocker
> Fix For: 3.0.0-alpha4
>
> Attachments: HDFS-11851.000.patch, HDFS-11851.001.patch,
> HDFS-11851.002.patch, HDFS-11851.003.patch, HDFS-11851.004.patch,
> HDFS-11851.005.patch
>
>
> HDFS-11529 introduced a deadlock into {{getGlobalJNIEnv()}} if an exception
> is thrown. {{getGlobalJNIEnv()}} holds {{jvmMutex}}, but
> {{printExceptionAndFree()}} will eventually try to acquire that lock in
> {{setTLSExceptionStrings()}}.
> The exception might get caught from {{loadFileSystems}}:
> {code}
> jthr = invokeMethod(env, NULL, STATIC, NULL,
> "org/apache/hadoop/fs/FileSystem",
> "loadFileSystems", "()V");
> if (jthr) {
> printExceptionAndFree(env, jthr, PRINT_EXC_ALL,
> "loadFileSystems");
> }
> }
> {code}
> and here's the relevant parts of the stack trace from where I call this API
> in Impala, which uses {{libhdfs}}:
> {code}
> #0 __lll_lock_wait () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
> #1 0x74a8d657 in _L_lock_909 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #2 0x74a8d480 in __GI___pthread_mutex_lock (mutex=0x47ce960
> ) at ../nptl/pthread_mutex_lock.c:79
> #3 0x02f06056 in mutexLock (m=) at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/mutexes.c:28
> #4 0x02efe817 in setTLSExceptionStrings (rootCause=0x0,
> stackTrace=0x0) at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:581
> #5 0x02f065d7 in printExceptionAndFreeV (env=0x513c1e8,
> exc=0x508a8c0, noPrintFlags=, fmt=0x34349cf "loadFileSystems",
> ap=0x7fffb660)
> at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:183
> #6 0x02f0683d in printExceptionAndFree (env=,
> exc=, noPrintFlags=, fmt=)
> at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:213
> #7 0x02eff60f in getGlobalJNIEnv () at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:463
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12222) Add EC information to BlockLocation
[
https://issues.apache.org/jira/browse/HDFS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162329#comment-16162329
]
Andrew Wang commented on HDFS-1:
Hi Huafeng, thanks for working on this, I took a look at the 005 patch, a few
review comments:
* I recommend moving the BlockLocation[] format example to
FileSystem#getFileBlockLocations and FileContext#getFileBlockLocations, which
are public APIs. Then, we can link to it from the other code locations.
* Related, the javadoc links to TestDistributedFileSystemWithECFile don't work
since prod code doesn't depend on test, recommend instead doing the above
change and linking to the public method
* Recommend adding a link to FileSystem#getFileBlockLocations to
LocatedFileStatus#getBlockLocations.
* Recommend adding a format explanation for a single BlockLocation (replicated
and EC) to BlockLocation as well, along with the link to FileSystem and
FileContext.
Regarding this blurb:
{code}
* In HDFS implementation, the BlockLocation of returned LocatedFileStatus
* will have different formats for replicated and erasure coded file. Please
* refer to HDFS for more details.
{code}
I'd prefer we drop this in most places, since this format should be compatible
for downstreams, and it'll be well documented on the public API classes noted
above.
> Add EC information to BlockLocation
> ---
>
> Key: HDFS-1
> URL: https://issues.apache.org/jira/browse/HDFS-1
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Huafeng Wang
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-1.001.patch, HDFS-1.002.patch,
> HDFS-1.003.patch, HDFS-1.004.patch, HDFS-1.005.patch
>
>
> HDFS applications query block location information to compute splits. One
> example of this is FileInputFormat:
> https://github.com/apache/hadoop/blob/d4015f8628dd973c7433639451a9acc3e741d2a2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L346
> You see bits of code like this that calculate offsets as follows:
> {noformat}
> long bytesInThisBlock = blkLocations[startIndex].getOffset() +
> blkLocations[startIndex].getLength() - offset;
> {noformat}
> EC confuses this since the block locations include parity block locations as
> well, which are not part of the logical file length. This messes up the
> offset calculation and thus topology/caching information too.
> Applications can figure out what's a parity block by reading the EC policy
> and then parsing the schema, but it'd be a lot better if we exposed this more
> generically in BlockLocation instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11772) Ozone: KSM: set creationTime for volume/bucket/key
[ https://issues.apache.org/jira/browse/HDFS-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162399#comment-16162399 ] Yiqun Lin commented on HDFS-11772: -- I have add creation time for volume/bucket/key, more details can see HDFS-12231, HDFS-12230 and HDFS-11984. I think we can close this now. > Ozone: KSM: set creationTime for volume/bucket/key > -- > > Key: HDFS-11772 > URL: https://issues.apache.org/jira/browse/HDFS-11772 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh > Labels: ozoneMerge > > Returns the volume information from KSM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11772) Ozone: KSM: set creationTime for volume/bucket/key
[ https://issues.apache.org/jira/browse/HDFS-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162399#comment-16162399 ] Yiqun Lin edited comment on HDFS-11772 at 9/12/17 2:32 AM: --- I have added creation time for volume/bucket/key, more details can see HDFS-12231, HDFS-12230 and HDFS-11984. I think we can close this JIRA now. was (Author: linyiqun): I have add creation time for volume/bucket/key, more details can see HDFS-12231, HDFS-12230 and HDFS-11984. I think we can close this now. > Ozone: KSM: set creationTime for volume/bucket/key > -- > > Key: HDFS-11772 > URL: https://issues.apache.org/jira/browse/HDFS-11772 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh > Labels: ozoneMerge > > Returns the volume information from KSM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12414) Ensure to use CLI command to enable/disable erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-12414: - Attachment: (was: HDFS-12414.002.patch) > Ensure to use CLI command to enable/disable erasure coding policy > - > > Key: HDFS-12414 > URL: https://issues.apache.org/jira/browse/HDFS-12414 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: SammiChen > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12414.001.patch > > > Currently, there are two methods for user to enable/disable a erasure coding > policy. One is through "dfs.namenode.ec.policies.enabled" property which is a > static way to configure the enabled erasure coding policies. Another is > through "enableErasureCodingPolicy" or "disabledErasureCodingPolicy" API > which can enabled or disable erasure coding policy at runtime. > When Namenode restart, there is potential state conflicts between the policy > defined in "dfs.namenode.ec.policies.enabled" and policy saved in fsImage. To > resolve the conflict and simplify the operation, it's better to use just one > way and remove the old method configuring the property. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12414) Ensure to use CLI command to enable/disable erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-12414: - Attachment: HDFS-12414.002.patch > Ensure to use CLI command to enable/disable erasure coding policy > - > > Key: HDFS-12414 > URL: https://issues.apache.org/jira/browse/HDFS-12414 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: SammiChen > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12414.001.patch, HDFS-12414.002.patch > > > Currently, there are two methods for user to enable/disable a erasure coding > policy. One is through "dfs.namenode.ec.policies.enabled" property which is a > static way to configure the enabled erasure coding policies. Another is > through "enableErasureCodingPolicy" or "disabledErasureCodingPolicy" API > which can enabled or disable erasure coding policy at runtime. > When Namenode restart, there is potential state conflicts between the policy > defined in "dfs.namenode.ec.policies.enabled" and policy saved in fsImage. To > resolve the conflict and simplify the operation, it's better to use just one > way and remove the old method configuring the property. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12420) Disable Namenode format when data already exists
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HDFS-12420:
--
Summary: Disable Namenode format when data already exists (was: Disallow
Namenode format when data already exists)
> Disable Namenode format when data already exists
>
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12420.01.patch
>
>
> Disallow NameNode format to avoid accidental formatting of Namenode in
> production cluster. If someone really wants to delete the complete fsImage
> they can first delete the metadata dir and then run {code} hdfs namenode
> -format{code} manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12420) Disable Namenode format when data already exists
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HDFS-12420:
--
Description: Disable NameNode format to avoid accidental formatting of
Namenode in production cluster. If someone really wants to delete the complete
fsImage, they can first delete the metadata dir and then run {code} hdfs
namenode -format{code} manually. (was: Disallow NameNode format to avoid
accidental formatting of Namenode in production cluster. If someone really
wants to delete the complete fsImage they can first delete the metadata dir and
then run {code} hdfs namenode -format{code} manually.)
> Disable Namenode format when data already exists
>
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12420.01.patch
>
>
> Disable NameNode format to avoid accidental formatting of Namenode in
> production cluster. If someone really wants to delete the complete fsImage,
> they can first delete the metadata dir and then run {code} hdfs namenode
> -format{code} manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12423) Ozone: TopN container choosing policy should ignore unnecessary containers
[
https://issues.apache.org/jira/browse/HDFS-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162443#comment-16162443
]
Weiwei Yang commented on HDFS-12423:
Thanks for filing this [~linyiqun], this is a good improvement to avoid
creating unnecessary tasks. +1 on the idea. Thanks.
> Ozone: TopN container choosing policy should ignore unnecessary containers
> --
>
> Key: HDFS-12423
> URL: https://issues.apache.org/jira/browse/HDFS-12423
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>
> TopN container choosing policy should ignore unnecessary containers.
> Currently TopN policy selects specified count of containers but not check the
> number of pending deletion blocks. So there is a chance there will be some
> unnecessary containers being chosen. That is say we will choose some
> containers that don't include any pending deletion blocks. The related output:
> {noformat}
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion,
> pending deletion blocks num: 1.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion,
> pending deletion blocks num: 0.
> {noformat}
> We should ignore these containers which pending deletion blocks num is 0,
> this can reduce some unnecessary block-deletion iterations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12414) Ensure to use CLI command to enable/disable erasure coding policy
[
https://issues.apache.org/jira/browse/HDFS-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162497#comment-16162497
]
Hadoop QA commented on HDFS-12414:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 44 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 1116 unchanged - 1 fixed = 1116 total (was 1117) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
48s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 58s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}118m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 |
| | hadoop.hdfs.server.namenode.TestReencryptionHandler |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestPipelines |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12414 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886569/HDFS-12414.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 3760804fc9a9 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
[jira] [Commented] (HDFS-11600) Refactor TestDFSStripedOutputStreamWithFailure test classes
[
https://issues.apache.org/jira/browse/HDFS-11600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162122#comment-16162122
]
Andrew Wang commented on HDFS-11600:
I posted an alternative disable patch on HDFS-12417.
I don't think we can do that much better when it comes to refactoring. We can
parameterize using JUnit instead of the classname and add better docs, but I
don't know how to get around having all these different test classes.
> Refactor TestDFSStripedOutputStreamWithFailure test classes
> ---
>
> Key: HDFS-11600
> URL: https://issues.apache.org/jira/browse/HDFS-11600
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Affects Versions: 3.0.0-alpha2
>Reporter: Andrew Wang
>Assignee: Elek, Marton
>Priority: Minor
> Attachments: HDFS-11600.002.patch, HDFS-11600-1.patch
>
>
> TestDFSStripedOutputStreamWithFailure has a great number of subclasses. The
> tests are parameterized based on the name of these subclasses.
> Seems like we could parameterize these tests with JUnit and then not need all
> these separate test classes.
> Another note, the tests will randomly return instead of running the test.
> Using {{Assume}} instead would make it more clear in the test output that
> these tests were skipped.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12407) Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or JournalNodeRpcServer fails to start
[
https://issues.apache.org/jira/browse/HDFS-12407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162123#comment-16162123
]
Arpit Agarwal commented on HDFS-12407:
--
Thanks [~ajayydv]!
+1 pending Jenkins.
> Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or
> JournalNodeRpcServer fails to start
> ---
>
> Key: HDFS-12407
> URL: https://issues.apache.org/jira/browse/HDFS-12407
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12407.01.patch, HDFS-12407.02.patch,
> HDFS-12407.03.patch
>
>
> Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or
> JournalNodeRpcServer fails to start.
> Steps to recreate the issue:
> # Change the http port for JournalNodeHttpServerr to some port which is
> already in use
> {code}dfs.journalnode.http-address{code}
> # Start the journalnode. JournalNodeHttpServer start will fail with bind
> exception while journalnode process will continue to run.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12406) dfsadmin command prints "Exception encountered" even if there is no exception, when debug is enabled
[
https://issues.apache.org/jira/browse/HDFS-12406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-12406:
-
Fix Version/s: 3.1.0
> dfsadmin command prints "Exception encountered" even if there is no
> exception, when debug is enabled
> -
>
> Key: HDFS-12406
> URL: https://issues.apache.org/jira/browse/HDFS-12406
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Nandakumar
>Assignee: Nandakumar
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HDFS-12406.000.patch
>
>
> In DFSAdmin we are printing {{"Exception encountered"}} at debug level for
> all the calls even if there is no exception.
> {code:title=DFSAdmin#run}
> if (LOG.isDebugEnabled()) {
> LOG.debug("Exception encountered:", debugException);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12417) Disable flaky TestDFSStripedOutputStreamWithFailure
[
https://issues.apache.org/jira/browse/HDFS-12417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162278#comment-16162278
]
Hadoop QA commented on HDFS-12417:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 46s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestFileCorruption |
| | hadoop.hdfs.TestDistributedFileSystem |
| | hadoop.hdfs.TestReplaceDatanodeOnFailure |
| | hadoop.hdfs.TestReconstructStripedFile |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12417 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886510/HDFS-12417.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5dc7c0dfe4de 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 738c2a9 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/21078/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/21078/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/21078/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was
[jira] [Commented] (HDFS-12098) Ozone: Datanode is unable to register with scm if scm starts later
[
https://issues.apache.org/jira/browse/HDFS-12098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162330#comment-16162330
]
Weiwei Yang commented on HDFS-12098:
Hi [~anu], [~vagarychen]
Thanks for revisiting this, I could not reproduce this either on latest code
base, looks like this was fixed by some other patches. This seems no longer a
valid issue, I think we can close it. Thanks for spending time trying to
reproduce this.
> Ozone: Datanode is unable to register with scm if scm starts later
> --
>
> Key: HDFS-12098
> URL: https://issues.apache.org/jira/browse/HDFS-12098
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, ozone, scm
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Labels: ozoneMerge
> Fix For: HDFS-7240
>
> Attachments: disabled-scm-test.patch, HDFS-12098-HDFS-7240.001.patch,
> HDFS-12098-HDFS-7240.002.patch, HDFS-12098-HDFS-7240.testcase-1.patch,
> HDFS-12098-HDFS-7240.testcase.patch, Screen Shot 2017-07-11 at 4.58.08
> PM.png, thread_dump.log
>
>
> Reproducing steps
> 1. Start namenode
> {{./bin/hdfs --daemon start namenode}}
> 2. Start datanode
> {{./bin/hdfs datanode}}
> will see following connection issues
> {noformat}
> 17/07/13 21:16:48 INFO ipc.Client: Retrying connect to server:
> ozone1.fyre.ibm.com/172.16.165.133:9861. Already tried 0 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> 17/07/13 21:16:49 INFO ipc.Client: Retrying connect to server:
> ozone1.fyre.ibm.com/172.16.165.133:9861. Already tried 1 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> 17/07/13 21:16:50 INFO ipc.Client: Retrying connect to server:
> ozone1.fyre.ibm.com/172.16.165.133:9861. Already tried 2 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> 17/07/13 21:16:51 INFO ipc.Client: Retrying connect to server:
> ozone1.fyre.ibm.com/172.16.165.133:9861. Already tried 3 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> {noformat}
> this is expected because scm is not started yet
> 3. Start scm
> {{./bin/hdfs scm}}
> expecting datanode can register to this scm, expecting the log in scm
> {noformat}
> 17/07/13 21:22:30 INFO node.SCMNodeManager: Data node with ID:
> af22862d-aafa-4941-9073-53224ae43e2c Registered.
> {noformat}
> but did *NOT* see this log. (_I debugged into the code and found the datanode
> state was transited SHUTDOWN unexpectedly because the thread leaks, each of
> those threads counted to set to next state and they all set to SHUTDOWN
> state_)
> 4. Create a container from scm CLI
> {{./bin/hdfs scm -container -create -c 20170714c0}}
> this fails with following exception
> {noformat}
> Creating container : 20170714c0.
> Error executing
> command:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.scm.exceptions.SCMException):
> Unable to create container while in chill mode
> at
> org.apache.hadoop.ozone.scm.container.ContainerMapping.allocateContainer(ContainerMapping.java:241)
> at
> org.apache.hadoop.ozone.scm.StorageContainerManager.allocateContainer(StorageContainerManager.java:392)
> at
> org.apache.hadoop.ozone.protocolPB.StorageContainerLocationProtocolServerSideTranslatorPB.allocateContainer(StorageContainerLocationProtocolServerSideTranslatorPB.java:73)
> {noformat}
> datanode was not registered to scm, thus it's still in chill mode.
> *Note*, if we start scm first, there is no such issue, I can create container
> from CLI without any problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12098) Ozone: Datanode is unable to register with scm if scm starts later
[
https://issues.apache.org/jira/browse/HDFS-12098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12098:
---
Resolution: Cannot Reproduce
Status: Resolved (was: Patch Available)
> Ozone: Datanode is unable to register with scm if scm starts later
> --
>
> Key: HDFS-12098
> URL: https://issues.apache.org/jira/browse/HDFS-12098
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, ozone, scm
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Labels: ozoneMerge
> Attachments: disabled-scm-test.patch, HDFS-12098-HDFS-7240.001.patch,
> HDFS-12098-HDFS-7240.002.patch, HDFS-12098-HDFS-7240.testcase-1.patch,
> HDFS-12098-HDFS-7240.testcase.patch, Screen Shot 2017-07-11 at 4.58.08
> PM.png, thread_dump.log
>
>
> Reproducing steps
> 1. Start namenode
> {{./bin/hdfs --daemon start namenode}}
> 2. Start datanode
> {{./bin/hdfs datanode}}
> will see following connection issues
> {noformat}
> 17/07/13 21:16:48 INFO ipc.Client: Retrying connect to server:
> ozone1.fyre.ibm.com/172.16.165.133:9861. Already tried 0 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> 17/07/13 21:16:49 INFO ipc.Client: Retrying connect to server:
> ozone1.fyre.ibm.com/172.16.165.133:9861. Already tried 1 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> 17/07/13 21:16:50 INFO ipc.Client: Retrying connect to server:
> ozone1.fyre.ibm.com/172.16.165.133:9861. Already tried 2 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> 17/07/13 21:16:51 INFO ipc.Client: Retrying connect to server:
> ozone1.fyre.ibm.com/172.16.165.133:9861. Already tried 3 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> {noformat}
> this is expected because scm is not started yet
> 3. Start scm
> {{./bin/hdfs scm}}
> expecting datanode can register to this scm, expecting the log in scm
> {noformat}
> 17/07/13 21:22:30 INFO node.SCMNodeManager: Data node with ID:
> af22862d-aafa-4941-9073-53224ae43e2c Registered.
> {noformat}
> but did *NOT* see this log. (_I debugged into the code and found the datanode
> state was transited SHUTDOWN unexpectedly because the thread leaks, each of
> those threads counted to set to next state and they all set to SHUTDOWN
> state_)
> 4. Create a container from scm CLI
> {{./bin/hdfs scm -container -create -c 20170714c0}}
> this fails with following exception
> {noformat}
> Creating container : 20170714c0.
> Error executing
> command:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.scm.exceptions.SCMException):
> Unable to create container while in chill mode
> at
> org.apache.hadoop.ozone.scm.container.ContainerMapping.allocateContainer(ContainerMapping.java:241)
> at
> org.apache.hadoop.ozone.scm.StorageContainerManager.allocateContainer(StorageContainerManager.java:392)
> at
> org.apache.hadoop.ozone.protocolPB.StorageContainerLocationProtocolServerSideTranslatorPB.allocateContainer(StorageContainerLocationProtocolServerSideTranslatorPB.java:73)
> {noformat}
> datanode was not registered to scm, thus it's still in chill mode.
> *Note*, if we start scm first, there is no such issue, I can create container
> from CLI without any problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12098) Ozone: Datanode is unable to register with scm if scm starts later
[
https://issues.apache.org/jira/browse/HDFS-12098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12098:
---
Affects Version/s: HDFS-7240
> Ozone: Datanode is unable to register with scm if scm starts later
> --
>
> Key: HDFS-12098
> URL: https://issues.apache.org/jira/browse/HDFS-12098
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, ozone, scm
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Labels: ozoneMerge
> Fix For: HDFS-7240
>
> Attachments: disabled-scm-test.patch, HDFS-12098-HDFS-7240.001.patch,
> HDFS-12098-HDFS-7240.002.patch, HDFS-12098-HDFS-7240.testcase-1.patch,
> HDFS-12098-HDFS-7240.testcase.patch, Screen Shot 2017-07-11 at 4.58.08
> PM.png, thread_dump.log
>
>
> Reproducing steps
> 1. Start namenode
> {{./bin/hdfs --daemon start namenode}}
> 2. Start datanode
> {{./bin/hdfs datanode}}
> will see following connection issues
> {noformat}
> 17/07/13 21:16:48 INFO ipc.Client: Retrying connect to server:
> ozone1.fyre.ibm.com/172.16.165.133:9861. Already tried 0 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> 17/07/13 21:16:49 INFO ipc.Client: Retrying connect to server:
> ozone1.fyre.ibm.com/172.16.165.133:9861. Already tried 1 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> 17/07/13 21:16:50 INFO ipc.Client: Retrying connect to server:
> ozone1.fyre.ibm.com/172.16.165.133:9861. Already tried 2 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> 17/07/13 21:16:51 INFO ipc.Client: Retrying connect to server:
> ozone1.fyre.ibm.com/172.16.165.133:9861. Already tried 3 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> {noformat}
> this is expected because scm is not started yet
> 3. Start scm
> {{./bin/hdfs scm}}
> expecting datanode can register to this scm, expecting the log in scm
> {noformat}
> 17/07/13 21:22:30 INFO node.SCMNodeManager: Data node with ID:
> af22862d-aafa-4941-9073-53224ae43e2c Registered.
> {noformat}
> but did *NOT* see this log. (_I debugged into the code and found the datanode
> state was transited SHUTDOWN unexpectedly because the thread leaks, each of
> those threads counted to set to next state and they all set to SHUTDOWN
> state_)
> 4. Create a container from scm CLI
> {{./bin/hdfs scm -container -create -c 20170714c0}}
> this fails with following exception
> {noformat}
> Creating container : 20170714c0.
> Error executing
> command:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.scm.exceptions.SCMException):
> Unable to create container while in chill mode
> at
> org.apache.hadoop.ozone.scm.container.ContainerMapping.allocateContainer(ContainerMapping.java:241)
> at
> org.apache.hadoop.ozone.scm.StorageContainerManager.allocateContainer(StorageContainerManager.java:392)
> at
> org.apache.hadoop.ozone.protocolPB.StorageContainerLocationProtocolServerSideTranslatorPB.allocateContainer(StorageContainerLocationProtocolServerSideTranslatorPB.java:73)
> {noformat}
> datanode was not registered to scm, thus it's still in chill mode.
> *Note*, if we start scm first, there is no such issue, I can create container
> from CLI without any problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12098) Ozone: Datanode is unable to register with scm if scm starts later
[
https://issues.apache.org/jira/browse/HDFS-12098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12098:
---
Fix Version/s: HDFS-7240
> Ozone: Datanode is unable to register with scm if scm starts later
> --
>
> Key: HDFS-12098
> URL: https://issues.apache.org/jira/browse/HDFS-12098
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, ozone, scm
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Labels: ozoneMerge
> Fix For: HDFS-7240
>
> Attachments: disabled-scm-test.patch, HDFS-12098-HDFS-7240.001.patch,
> HDFS-12098-HDFS-7240.002.patch, HDFS-12098-HDFS-7240.testcase-1.patch,
> HDFS-12098-HDFS-7240.testcase.patch, Screen Shot 2017-07-11 at 4.58.08
> PM.png, thread_dump.log
>
>
> Reproducing steps
> 1. Start namenode
> {{./bin/hdfs --daemon start namenode}}
> 2. Start datanode
> {{./bin/hdfs datanode}}
> will see following connection issues
> {noformat}
> 17/07/13 21:16:48 INFO ipc.Client: Retrying connect to server:
> ozone1.fyre.ibm.com/172.16.165.133:9861. Already tried 0 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> 17/07/13 21:16:49 INFO ipc.Client: Retrying connect to server:
> ozone1.fyre.ibm.com/172.16.165.133:9861. Already tried 1 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> 17/07/13 21:16:50 INFO ipc.Client: Retrying connect to server:
> ozone1.fyre.ibm.com/172.16.165.133:9861. Already tried 2 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> 17/07/13 21:16:51 INFO ipc.Client: Retrying connect to server:
> ozone1.fyre.ibm.com/172.16.165.133:9861. Already tried 3 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> {noformat}
> this is expected because scm is not started yet
> 3. Start scm
> {{./bin/hdfs scm}}
> expecting datanode can register to this scm, expecting the log in scm
> {noformat}
> 17/07/13 21:22:30 INFO node.SCMNodeManager: Data node with ID:
> af22862d-aafa-4941-9073-53224ae43e2c Registered.
> {noformat}
> but did *NOT* see this log. (_I debugged into the code and found the datanode
> state was transited SHUTDOWN unexpectedly because the thread leaks, each of
> those threads counted to set to next state and they all set to SHUTDOWN
> state_)
> 4. Create a container from scm CLI
> {{./bin/hdfs scm -container -create -c 20170714c0}}
> this fails with following exception
> {noformat}
> Creating container : 20170714c0.
> Error executing
> command:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.scm.exceptions.SCMException):
> Unable to create container while in chill mode
> at
> org.apache.hadoop.ozone.scm.container.ContainerMapping.allocateContainer(ContainerMapping.java:241)
> at
> org.apache.hadoop.ozone.scm.StorageContainerManager.allocateContainer(StorageContainerManager.java:392)
> at
> org.apache.hadoop.ozone.protocolPB.StorageContainerLocationProtocolServerSideTranslatorPB.allocateContainer(StorageContainerLocationProtocolServerSideTranslatorPB.java:73)
> {noformat}
> datanode was not registered to scm, thus it's still in chill mode.
> *Note*, if we start scm first, there is no such issue, I can create container
> from CLI without any problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12414) Ensure to use CLI command to enable/disable erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-12414: - Attachment: HDFS-12414.002.patch 1. fix style issues 2. fix failed unit tests > Ensure to use CLI command to enable/disable erasure coding policy > - > > Key: HDFS-12414 > URL: https://issues.apache.org/jira/browse/HDFS-12414 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: SammiChen > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12414.001.patch, HDFS-12414.002.patch > > > Currently, there are two methods for user to enable/disable a erasure coding > policy. One is through "dfs.namenode.ec.policies.enabled" property which is a > static way to configure the enabled erasure coding policies. Another is > through "enableErasureCodingPolicy" or "disabledErasureCodingPolicy" API > which can enabled or disable erasure coding policy at runtime. > When Namenode restart, there is potential state conflicts between the policy > defined in "dfs.namenode.ec.policies.enabled" and policy saved in fsImage. To > resolve the conflict and simplify the operation, it's better to use just one > way and remove the old method configuring the property. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-11772) Ozone: KSM: set creationTime for volume/bucket/key
[ https://issues.apache.org/jira/browse/HDFS-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDFS-11772. - Resolution: Duplicate > Ozone: KSM: set creationTime for volume/bucket/key > -- > > Key: HDFS-11772 > URL: https://issues.apache.org/jira/browse/HDFS-11772 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh > Labels: ozoneMerge > > Returns the volume information from KSM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12389) Ozone: oz commandline list calls should return valid JSON format output
[
https://issues.apache.org/jira/browse/HDFS-12389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162432#comment-16162432
]
Weiwei Yang commented on HDFS-12389:
Thanks for the +1 [~anu], I am going to commit this shortly. Thank you.
> Ozone: oz commandline list calls should return valid JSON format output
> ---
>
> Key: HDFS-12389
> URL: https://issues.apache.org/jira/browse/HDFS-12389
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Labels: ozoneMerge
> Attachments: HDFS-12389-HDFS-7240.001.patch,
> HDFS-12389-HDFS-7240.002.patch, json_output_test.log
>
>
> At present the outputs of {{listVolume}}, {{listBucket}} and {{listKey}} are
> hard to parse, for example following call
> {code}
> ./bin/hdfs oz -listVolume http://localhost:9864 -user wwei
> {code}
> lists all volumes in my cluster and it returns
> {noformat}
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "modifiedOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "size" : 10240,
> "keyName" : "key-0-22381",
> "dataFileName" : null
> }
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "modifiedOn" : "Mon, 04 Sep 2017 03:25:22 GMT",
> "size" : 10240,
> "keyName" : "key-0-22381",
> "dataFileName" : null
> }
> ...
> {noformat}
> this is not a valid JSON format output hence it is hard to parse in clients'
> script for further interactions. Propose to reformat them to valid JSON data.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11676) Ozone: SCM CLI: Implement close container command
[
https://issues.apache.org/jira/browse/HDFS-11676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-11676:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-7240
Status: Resolved (was: Patch Available)
I've committed this to the feature branch, thanks [~vagarychen] for the patch
and [~linyiqun] for the review.
> Ozone: SCM CLI: Implement close container command
> -
>
> Key: HDFS-11676
> URL: https://issues.apache.org/jira/browse/HDFS-11676
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Chen Liang
> Labels: OzonePostMerge, tocheck
> Fix For: HDFS-7240
>
> Attachments: HDFS-11676-HDFS-7240.001.patch,
> HDFS-11676-HDFS-7240.002.patch, HDFS-11676-HDFS-7240.003.patch
>
>
> Implement close container command
> {code}
> hdfs scm -container close
> {code}
> This command connects to SCM and closes a container. Once the container is
> closed in the SCM, the corresponding container is closed at the appropriate
> datanode. if the container does not exist, it will return an error.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12423) Ozone: TopN container choosing policy should ignore unnecessary containers
Yiqun Lin created HDFS-12423:
Summary: Ozone: TopN container choosing policy should ignore
unnecessary containers
Key: HDFS-12423
URL: https://issues.apache.org/jira/browse/HDFS-12423
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Affects Versions: HDFS-7240
Reporter: Yiqun Lin
Assignee: Yiqun Lin
TopN container choosing policy should ignore unnecessary containers. Currently
TopN policy selects specified count of containers but not check the number of
pending deletion blocks. So there is a chance there will be some unnecessary
containers being chosen. That is say we will choose some containers that don't
include any pending deletion blocks. The related output:
{noformat}
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion, pending
deletion blocks num: 1.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion, pending
deletion blocks num: 0.
{noformat}
We should ignore these containers which pending deletion blocks num is 0, this
can reduce some unnecessary iterations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12414) Ensure to use CLI command to enable/disable erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-12414: - Status: Patch Available (was: Open) > Ensure to use CLI command to enable/disable erasure coding policy > - > > Key: HDFS-12414 > URL: https://issues.apache.org/jira/browse/HDFS-12414 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: SammiChen > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12414.001.patch, HDFS-12414.002.patch > > > Currently, there are two methods for user to enable/disable a erasure coding > policy. One is through "dfs.namenode.ec.policies.enabled" property which is a > static way to configure the enabled erasure coding policies. Another is > through "enableErasureCodingPolicy" or "disabledErasureCodingPolicy" API > which can enabled or disable erasure coding policy at runtime. > When Namenode restart, there is potential state conflicts between the policy > defined in "dfs.namenode.ec.policies.enabled" and policy saved in fsImage. To > resolve the conflict and simplify the operation, it's better to use just one > way and remove the old method configuring the property. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12409) Add metrics of execution time of different stages in EC recovery task
[
https://issues.apache.org/jira/browse/HDFS-12409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162331#comment-16162331
]
Hadoop QA commented on HDFS-12409:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 42s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
22s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}138m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.TestFileAppendRestart |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestGetBlocks |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicy |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.TestModTime |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.server.namenode.TestReencryption |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
[jira] [Commented] (HDFS-12420) Disallow Namenode format when data already exists
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162364#comment-16162364
]
Hadoop QA commented on HDFS-12420:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestSaveNamespace |
| | hadoop.hdfs.qjournal.TestNNWithQJM |
| | hadoop.hdfs.TestFileAppendRestart |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.server.namenode.TestNameEditsConfigs |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 |
| | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.server.namenode.TestGenericJournalConf |
| | hadoop.hdfs.TestAclsEndToEnd |
| | hadoop.hdfs.TestDistributedFileSystem |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | hadoop.hdfs.server.namenode.TestClusterId |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
| | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| | org.apache.hadoop.hdfs.server.namenode.TestEditLogRace |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12420 |
| JIRA Patch URL |
[jira] [Updated] (HDFS-12328) Ozone: Purge metadata of deleted blocks after max retry times
[
https://issues.apache.org/jira/browse/HDFS-12328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12328:
---
Priority: Major (was: Critical)
> Ozone: Purge metadata of deleted blocks after max retry times
> -
>
> Key: HDFS-12328
> URL: https://issues.apache.org/jira/browse/HDFS-12328
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yuanbo Liu
>Assignee: Yuanbo Liu
> Labels: OzonePostMerge
>
> In HDFS-12283, we set the value of count to -1 if blocks cannot be deleted
> after max retry times. We need to provide APIs for admins to purge the "-1"
> metadata manually. Implement these commands:
> list the txids
> {code}
> hdfs scm -txid list -count -retry
> {code}
> delete the txid
> {code}
> hdfs scm -txid delete -id
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12416) BlockPlacementPolicyDefault will cause NN shutdown if log level is changed
[
https://issues.apache.org/jira/browse/HDFS-12416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suhan Mao updated HDFS-12416:
-
Attachment: HDFS-12416.001.patch
> BlockPlacementPolicyDefault will cause NN shutdown if log level is changed
> --
>
> Key: HDFS-12416
> URL: https://issues.apache.org/jira/browse/HDFS-12416
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
>Affects Versions: 2.7.4, 3.0.0-alpha3
>Reporter: Suhan Mao
> Attachments: HDFS-12416.001.patch, HDFS-12416.patch
>
> Original Estimate: 5h
> Remaining Estimate: 5h
>
> In BlockPlacementPolicyDefault.chooseRandom method.
> The code are in below structure:
> {code:java}
> StringBuilder builder = null;
> if (LOG.isDebugEnabled()) {
> builder = debugLoggingBuilder.get();
> builder.setLength(0);
> builder.append("[");
> }
> while(numOfReplicas > 0){
> chooseDataNode(scope, excludedNodes)
> if (LOG.isDebugEnabled()) {
> builder.append("\nNode ").append(NodeBase.getPath(chosenNode))
> .append(" [");
> }
> }
> {code}
> There's a possibility that the loglevel is INFO before entering while loop,
> but the loglevel is changed to DEBUG inside the loop through web UI.
> In that case, builder is not initialized in the beginning and
> NullPointerException will throw and this will cause NN exiting.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12415) Ozone: TestXceiverClientManager occasionally fails
[
https://issues.apache.org/jira/browse/HDFS-12415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161063#comment-16161063
]
Hadoop QA commented on HDFS-12415:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} hadoop-hdfs-project_hadoop-hdfs generated 0 new +
428 unchanged - 1 fixed = 428 total (was 429) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}128m 15s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestFileAppendRestart |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.TestGenericRefresh |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 |
| | hadoop.ozone.container.common.TestDatanodeStateMachine |
| | hadoop.ozone.TestMiniOzoneCluster |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestHDFSFileSystemContract |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | hadoop.ozone.scm.TestAllocateContainer |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12415 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886367/HDFS-12415-HDFS-7240.001.patch
|
| Optional Tests |
[jira] [Updated] (HDFS-12416) BlockPlacementPolicyDefault will cause NN shutdown if log level is changed
[
https://issues.apache.org/jira/browse/HDFS-12416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suhan Mao updated HDFS-12416:
-
Status: Patch Available (was: Open)
Patch submitted.
The patch will check whether the stringbuilder is null before it is used.
> BlockPlacementPolicyDefault will cause NN shutdown if log level is changed
> --
>
> Key: HDFS-12416
> URL: https://issues.apache.org/jira/browse/HDFS-12416
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
>Affects Versions: 3.0.0-alpha3, 2.7.4
>Reporter: Suhan Mao
> Attachments: HDFS-12416.001.patch, HDFS-12416.patch
>
> Original Estimate: 5h
> Remaining Estimate: 5h
>
> In BlockPlacementPolicyDefault.chooseRandom method.
> The code are in below structure:
> {code:java}
> StringBuilder builder = null;
> if (LOG.isDebugEnabled()) {
> builder = debugLoggingBuilder.get();
> builder.setLength(0);
> builder.append("[");
> }
> while(numOfReplicas > 0){
> .
> chooseDataNode(scope, excludedNodes)
> .
> if (LOG.isDebugEnabled()) {
> builder.append("\nNode ").append(NodeBase.getPath(chosenNode))
> .append(" [");
> }
> }
> {code}
> There's a possibility that the loglevel is INFO before entering while loop,
> but the loglevel is changed to DEBUG inside the loop through web UI.
> In that case, builder is not initialized in the beginning and
> NullPointerException will throw and this will cause NN exiting.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12008) Improve the available-space block placement policy
[
https://issues.apache.org/jira/browse/HDFS-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160883#comment-16160883
]
Hadoop QA commented on HDFS-12008:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
58s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 28 unchanged - 2 fixed = 30 total (was 30) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}132m 15s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}162m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicy |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.web.TestWebHdfsWithRestCsrfPreventionFilter |
| | hadoop.hdfs.server.blockmanagement.TestBlockManager |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup |
| | hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer |
| |
[jira] [Created] (HDFS-12416) BlockPlacementPolicyDefault will cause NN shutdown if log level is changed
Suhan Mao created HDFS-12416:
Summary: BlockPlacementPolicyDefault will cause NN shutdown if log
level is changed
Key: HDFS-12416
URL: https://issues.apache.org/jira/browse/HDFS-12416
Project: Hadoop HDFS
Issue Type: Bug
Components: block placement
Affects Versions: 3.0.0-alpha3, 2.7.4
Reporter: Suhan Mao
In BlockPlacementPolicyDefault.chooseRandom method.
The code are in below structure:
{code:java}
StringBuilder builder = null;
if (LOG.isDebugEnabled()) {
builder = debugLoggingBuilder.get();
builder.setLength(0);
builder.append("[");
}
while(numOfReplicas > 0){
chooseDataNode(scope, excludedNodes)
if (LOG.isDebugEnabled()) {
builder.append("\nNode ").append(NodeBase.getPath(chosenNode))
.append(" [");
}
}
{code}
There's a possibility that the loglevel is INFO before entering while loop, but
the loglevel is changed to DEBUG inside the loop through web UI.
In that case, builder is not initialized in the beginning and
NullPointerException will throw and this will cause NN exiting.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12416) BlockPlacementPolicyDefault will cause NN shutdown if log level is changed
[
https://issues.apache.org/jira/browse/HDFS-12416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suhan Mao updated HDFS-12416:
-
Description:
In BlockPlacementPolicyDefault.chooseRandom method.
The code are in below structure:
{code:java}
StringBuilder builder = null;
if (LOG.isDebugEnabled()) {
builder = debugLoggingBuilder.get();
builder.setLength(0);
builder.append("[");
}
while(numOfReplicas > 0){
.
chooseDataNode(scope, excludedNodes)
.
if (LOG.isDebugEnabled()) {
builder.append("\nNode ").append(NodeBase.getPath(chosenNode))
.append(" [");
}
}
{code}
There's a possibility that the loglevel is INFO before entering while loop, but
the loglevel is changed to DEBUG inside the loop through web UI.
In that case, builder is not initialized in the beginning and
NullPointerException will throw and this will cause NN exiting.
was:
In BlockPlacementPolicyDefault.chooseRandom method.
The code are in below structure:
{code:java}
StringBuilder builder = null;
if (LOG.isDebugEnabled()) {
builder = debugLoggingBuilder.get();
builder.setLength(0);
builder.append("[");
}
while(numOfReplicas > 0){
chooseDataNode(scope, excludedNodes)
if (LOG.isDebugEnabled()) {
builder.append("\nNode ").append(NodeBase.getPath(chosenNode))
.append(" [");
}
}
{code}
There's a possibility that the loglevel is INFO before entering while loop, but
the loglevel is changed to DEBUG inside the loop through web UI.
In that case, builder is not initialized in the beginning and
NullPointerException will throw and this will cause NN exiting.
> BlockPlacementPolicyDefault will cause NN shutdown if log level is changed
> --
>
> Key: HDFS-12416
> URL: https://issues.apache.org/jira/browse/HDFS-12416
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
>Affects Versions: 2.7.4, 3.0.0-alpha3
>Reporter: Suhan Mao
> Attachments: HDFS-12416.001.patch, HDFS-12416.patch
>
> Original Estimate: 5h
> Remaining Estimate: 5h
>
> In BlockPlacementPolicyDefault.chooseRandom method.
> The code are in below structure:
> {code:java}
> StringBuilder builder = null;
> if (LOG.isDebugEnabled()) {
> builder = debugLoggingBuilder.get();
> builder.setLength(0);
> builder.append("[");
> }
> while(numOfReplicas > 0){
> .
> chooseDataNode(scope, excludedNodes)
> .
> if (LOG.isDebugEnabled()) {
> builder.append("\nNode ").append(NodeBase.getPath(chosenNode))
> .append(" [");
> }
> }
> {code}
> There's a possibility that the loglevel is INFO before entering while loop,
> but the loglevel is changed to DEBUG inside the loop through web UI.
> In that case, builder is not initialized in the beginning and
> NullPointerException will throw and this will cause NN exiting.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12415) Ozone: TestXceiverClientManager occasionally fails
[
https://issues.apache.org/jira/browse/HDFS-12415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12415:
---
Attachment: HDFS-12415-HDFS-7240.001.patch
> Ozone: TestXceiverClientManager occasionally fails
> --
>
> Key: HDFS-12415
> URL: https://issues.apache.org/jira/browse/HDFS-12415
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: HDFS-12415-HDFS-7240.001.patch
>
>
> TestXceiverClientManager seems to be occasionally failing in some jenkins
> jobs,
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.scm.node.SCMNodeManager.getNodeStat(SCMNodeManager.java:828)
> at
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.hasEnoughSpace(SCMCommonPolicy.java:147)
> at
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.lambda$chooseDatanodes$0(SCMCommonPolicy.java:125)
> {noformat}
> see more from [this
> report|https://builds.apache.org/job/PreCommit-HDFS-Build/21065/testReport/]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12415) Ozone: TestXceiverClientManager occasionally fails
[
https://issues.apache.org/jira/browse/HDFS-12415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12415:
---
Status: Patch Available (was: Open)
> Ozone: TestXceiverClientManager occasionally fails
> --
>
> Key: HDFS-12415
> URL: https://issues.apache.org/jira/browse/HDFS-12415
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: HDFS-12415-HDFS-7240.001.patch
>
>
> TestXceiverClientManager seems to be occasionally failing in some jenkins
> jobs,
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.scm.node.SCMNodeManager.getNodeStat(SCMNodeManager.java:828)
> at
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.hasEnoughSpace(SCMCommonPolicy.java:147)
> at
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.lambda$chooseDatanodes$0(SCMCommonPolicy.java:125)
> {noformat}
> see more from [this
> report|https://builds.apache.org/job/PreCommit-HDFS-Build/21065/testReport/]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12415) Ozone: TestXceiverClientManager occasionally fails
[
https://issues.apache.org/jira/browse/HDFS-12415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160899#comment-16160899
]
Weiwei Yang commented on HDFS-12415:
It looks like test cases are calling {{allocateContainer}} before ozone is
fully started, add {{cluster.waitOzoneReady()}} should be able to fix this.
Uploaded a simple patch to fix. Please help to review.
> Ozone: TestXceiverClientManager occasionally fails
> --
>
> Key: HDFS-12415
> URL: https://issues.apache.org/jira/browse/HDFS-12415
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
>
> TestXceiverClientManager seems to be occasionally failing in some jenkins
> jobs,
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.scm.node.SCMNodeManager.getNodeStat(SCMNodeManager.java:828)
> at
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.hasEnoughSpace(SCMCommonPolicy.java:147)
> at
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.lambda$chooseDatanodes$0(SCMCommonPolicy.java:125)
> {noformat}
> see more from [this
> report|https://builds.apache.org/job/PreCommit-HDFS-Build/21065/testReport/]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12414) Ensure to use CLI command to enable/disable erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-12414: - Attachment: HDFS-12414.001.patch Initial patch > Ensure to use CLI command to enable/disable erasure coding policy > - > > Key: HDFS-12414 > URL: https://issues.apache.org/jira/browse/HDFS-12414 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: SammiChen > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12414.001.patch > > > Currently, there are two methods for user to enable/disable a erasure coding > policy. One is through "dfs.namenode.ec.policies.enabled" property which is a > static way to configure the enabled erasure coding policies. Another is > through "enableErasureCodingPolicy" or "disabledErasureCodingPolicy" API > which can enabled or disable erasure coding policy at runtime. > When Namenode restart, there is potential state conflicts between the policy > defined in "dfs.namenode.ec.policies.enabled" and policy saved in fsImage. To > resolve the conflict and simplify the operation, it's better to use just one > way and remove the old method configuring the property. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12414) Ensure to use CLI command to enable/disable erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-12414: - Status: Patch Available (was: Open) > Ensure to use CLI command to enable/disable erasure coding policy > - > > Key: HDFS-12414 > URL: https://issues.apache.org/jira/browse/HDFS-12414 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: SammiChen > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12414.001.patch > > > Currently, there are two methods for user to enable/disable a erasure coding > policy. One is through "dfs.namenode.ec.policies.enabled" property which is a > static way to configure the enabled erasure coding policies. Another is > through "enableErasureCodingPolicy" or "disabledErasureCodingPolicy" API > which can enabled or disable erasure coding policy at runtime. > When Namenode restart, there is potential state conflicts between the policy > defined in "dfs.namenode.ec.policies.enabled" and policy saved in fsImage. To > resolve the conflict and simplify the operation, it's better to use just one > way and remove the old method configuring the property. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[
https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161004#comment-16161004
]
Weiwei Yang commented on HDFS-12370:
I have tested this patch in a cluster, here is the steps I have tried
1. Decrease the container size from 5G to 1MB, this way we can create a lot of
containers for testing.
2. Create 2000 keys using corona
{code}
./bin/hdfs corona -numOfThreads 10 -numOfVolumes 1 -numOfBuckets 1 -numOfKeys
2000
{code}
3. Check containers have been created, there were 20
{code}
[wwei@ozone1 containers]$ ls
1d163265-8d47-4ed3-845f-f7d3eb569b83 b0865025-acfe-48a0-a5b2-d36224ecdb20
21017018-be32-44e6-9f62-e7fc9f6a6021 bcdb31bb-beee-47d0-8734-e407e64789c5
4cccadc8-ef5e-466d-bd9e-5b9705f8748c c7a85e6e-3528-45d1-8063-cd7fec114545
55b11be9-f16f-4620-a310-07c9be3bbfee ce883b83-2543-4b81-bdaf-6813b817b889
684556ea-a668-4bd3-82ee-8836db6fcba5 d7e3a36c-c68b-454c-80ce-23e3e92f3b1a
73e1a9b5-b77a-4230-a148-86b7f945dd63 da94ea5d-a279-4955-9c09-726579a167ec
75885243-3ae7-4f55-82dc-475f3d98f1f3 e753d4e6-19a1-4344-a97c-853d63ca0f81
77d3a401-8ece-41f0-b8e6-88d071aee301 e98ccb38-6b1e-4dc7-ac15-0f985f678e1f
84fdce99-3f69-4600-a16c-fc4fee20da01 ece9d60a-d5c9-434c-95bf-b3c425693eca
8f888499-397a-4c24-86bd-18dc73d2f870 ff1075d6-37d2-4b70-aae9-9cb58839f9ea
{code}
4. Get all keys by listVolume, listBucket and listKey calls, and redirect all
keys to a json file
{code}
./bin/hdfs oz -listVolume http://localhost:9864 -user wwei
./bin/hdfs oz -listBucket http://localhost:9864/vol-0-70079 -user wwei
./bin/hdfs oz -listKey http://localhost:9864/vol-0-70079/bucket-0-62742 -user
wwei > /tmp/keys.json
{code}
5. Write a python script to parse {{keys.json}} and delete keys one by one
{code}
import json
import subprocess
with open('keys.json') as data_file:
data = json.load(data_file)
for key in data:
print key['keyName']
fullName="http://localhost:9864/vol-0-70079/bucket-0-62742/"+key['keyName']
print "deleting key " + fullName
subprocess.check_output(["bash",
"/home/wwei/hadoop-3.0.0-beta1-SNAPSHOT/bin/hdfs", "oz", "-deleteKey",
fullName, "-user", "wwei"])
{code}
6. From the log, it seems it scans containers according to the order of the
number of pending deletion keys.
{code}
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container e98ccb38-6b1e-4dc7-ac15-0f985f678e1f for block deletion, pending
deletion blocks num: 9.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion, pending
deletion blocks num: 1.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 684556ea-a668-4bd3-82ee-8836db6fcba5 for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 73e1a9b5-b77a-4230-a148-86b7f945dd63 for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 75885243-3ae7-4f55-82dc-475f3d98f1f3 for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 77d3a401-8ece-41f0-b8e6-88d071aee301 for block deletion, pending
deletion blocks num: 0.
...
17/09/11 03:00:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 77d3a401-8ece-41f0-b8e6-88d071aee301 for block deletion, pending
deletion blocks num: 4.
17/09/11 03:00:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container e98ccb38-6b1e-4dc7-ac15-0f985f678e1f for block deletion, pending
deletion blocks num: 4.
17/09/11 03:00:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion, pending
deletion blocks num: 1.
17/09/11 03:00:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion, pending
deletion blocks num: 0.
17/09/11 03:00:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 21017018-be32-44e6-9f62-e7fc9f6a6021 for
[jira] [Created] (HDFS-12415) Ozone: TestXceiverClientManager occasionally fails
Weiwei Yang created HDFS-12415:
--
Summary: Ozone: TestXceiverClientManager occasionally fails
Key: HDFS-12415
URL: https://issues.apache.org/jira/browse/HDFS-12415
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7240
Reporter: Weiwei Yang
Assignee: Weiwei Yang
Priority: Minor
TestXceiverClientManager seems to be occasionally failing in some jenkins jobs,
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.ozone.scm.node.SCMNodeManager.getNodeStat(SCMNodeManager.java:828)
at
org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.hasEnoughSpace(SCMCommonPolicy.java:147)
at
org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.lambda$chooseDatanodes$0(SCMCommonPolicy.java:125)
{noformat}
see more from [this
report|https://builds.apache.org/job/PreCommit-HDFS-Build/21065/testReport/]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[ https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12370: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Status: Resolved (was: Patch Available) I've committed this to the feature branch, thanks a lot for the contribution [~linyiqun]! > Ozone: Implement TopN container choosing policy for BlockDeletionService > > > Key: HDFS-12370 > URL: https://issues.apache.org/jira/browse/HDFS-12370 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12370-HDFS-7240.001.patch, > HDFS-12370-HDFS-7240.002.patch, HDFS-12370-HDFS-7240.003.patch > > > Implement TopN container choosing policy for BlockDeletionService. This is > discussed from HDFS-12354. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12416) BlockPlacementPolicyDefault will cause NN shutdown if log level is changed
[
https://issues.apache.org/jira/browse/HDFS-12416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suhan Mao updated HDFS-12416:
-
Attachment: HDFS-12416.patch
> BlockPlacementPolicyDefault will cause NN shutdown if log level is changed
> --
>
> Key: HDFS-12416
> URL: https://issues.apache.org/jira/browse/HDFS-12416
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
>Affects Versions: 2.7.4, 3.0.0-alpha3
>Reporter: Suhan Mao
> Attachments: HDFS-12416.patch
>
> Original Estimate: 5h
> Remaining Estimate: 5h
>
> In BlockPlacementPolicyDefault.chooseRandom method.
> The code are in below structure:
> {code:java}
> StringBuilder builder = null;
> if (LOG.isDebugEnabled()) {
> builder = debugLoggingBuilder.get();
> builder.setLength(0);
> builder.append("[");
> }
> while(numOfReplicas > 0){
> chooseDataNode(scope, excludedNodes)
> if (LOG.isDebugEnabled()) {
> builder.append("\nNode ").append(NodeBase.getPath(chosenNode))
> .append(" [");
> }
> }
> {code}
> There's a possibility that the loglevel is INFO before entering while loop,
> but the loglevel is changed to DEBUG inside the loop through web UI.
> In that case, builder is not initialized in the beginning and
> NullPointerException will throw and this will cause NN exiting.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12416) BlockPlacementPolicyDefault will cause NN shutdown if log level is changed
[
https://issues.apache.org/jira/browse/HDFS-12416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161083#comment-16161083
]
Hadoop QA commented on HDFS-12416:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-12416 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-12416 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886391/HDFS-12416.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/21069/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> BlockPlacementPolicyDefault will cause NN shutdown if log level is changed
> --
>
> Key: HDFS-12416
> URL: https://issues.apache.org/jira/browse/HDFS-12416
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
>Affects Versions: 2.7.4, 3.0.0-alpha3
>Reporter: Suhan Mao
> Attachments: HDFS-12416.001.patch, HDFS-12416.patch
>
> Original Estimate: 5h
> Remaining Estimate: 5h
>
> In BlockPlacementPolicyDefault.chooseRandom method.
> The code are in below structure:
> {code:java}
> StringBuilder builder = null;
> if (LOG.isDebugEnabled()) {
> builder = debugLoggingBuilder.get();
> builder.setLength(0);
> builder.append("[");
> }
> while(numOfReplicas > 0){
> .
> chooseDataNode(scope, excludedNodes)
> .
> if (LOG.isDebugEnabled()) {
> builder.append("\nNode ").append(NodeBase.getPath(chosenNode))
> .append(" [");
> }
> }
> {code}
> There's a possibility that the loglevel is INFO before entering while loop,
> but the loglevel is changed to DEBUG inside the loop through web UI.
> In that case, builder is not initialized in the beginning and
> NullPointerException will throw and this will cause NN exiting.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12416) BlockPlacementPolicyDefault will cause NN shutdown if log level is changed
[
https://issues.apache.org/jira/browse/HDFS-12416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161089#comment-16161089
]
Brahma Reddy Battula commented on HDFS-12416:
-
Dupe of HDFS-12177 and HDFS-11827..?
> BlockPlacementPolicyDefault will cause NN shutdown if log level is changed
> --
>
> Key: HDFS-12416
> URL: https://issues.apache.org/jira/browse/HDFS-12416
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
>Affects Versions: 2.7.4, 3.0.0-alpha3
>Reporter: Suhan Mao
> Attachments: HDFS-12416.001.patch, HDFS-12416.patch
>
> Original Estimate: 5h
> Remaining Estimate: 5h
>
> In BlockPlacementPolicyDefault.chooseRandom method.
> The code are in below structure:
> {code:java}
> StringBuilder builder = null;
> if (LOG.isDebugEnabled()) {
> builder = debugLoggingBuilder.get();
> builder.setLength(0);
> builder.append("[");
> }
> while(numOfReplicas > 0){
> .
> chooseDataNode(scope, excludedNodes)
> .
> if (LOG.isDebugEnabled()) {
> builder.append("\nNode ").append(NodeBase.getPath(chosenNode))
> .append(" [");
> }
> }
> {code}
> There's a possibility that the loglevel is INFO before entering while loop,
> but the loglevel is changed to DEBUG inside the loop through web UI.
> In that case, builder is not initialized in the beginning and
> NullPointerException will throw and this will cause NN exiting.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work stopped] (HDFS-12415) Ozone: TestXceiverClientManager occasionally fails
[
https://issues.apache.org/jira/browse/HDFS-12415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-12415 stopped by Weiwei Yang.
--
> Ozone: TestXceiverClientManager occasionally fails
> --
>
> Key: HDFS-12415
> URL: https://issues.apache.org/jira/browse/HDFS-12415
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: HDFS-12415-HDFS-7240.001.patch
>
>
> TestXceiverClientManager seems to be occasionally failing in some jenkins
> jobs,
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.scm.node.SCMNodeManager.getNodeStat(SCMNodeManager.java:828)
> at
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.hasEnoughSpace(SCMCommonPolicy.java:147)
> at
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.lambda$chooseDatanodes$0(SCMCommonPolicy.java:125)
> {noformat}
> see more from [this
> report|https://builds.apache.org/job/PreCommit-HDFS-Build/21065/testReport/]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12395) Support erasure coding policy operations in namenode edit log
[
https://issues.apache.org/jira/browse/HDFS-12395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161095#comment-16161095
]
Rakesh R commented on HDFS-12395:
-
Thanks [~Sammi] for the continuous efforts.
bq. I'm not very sure about the "Update javadocs" comments, is that because
new parameter "logRetryCache" is added for involved API?
Yes, good to could update javadoc,
(1) with new param.
{code}
* @param logRetryCache whether to record RPC ids in editlog for retry cache
* rebuilding
{code}
(2) Secondly, since you are touching {{FSDirErasureCodingOp}} functions, could
you please add javadoc for the APIs which doesn't have details presently.
{code}
FSDirErasureCodingOp#addErasureCodingPolicy,
FSDirErasureCodingOp#removeErasureCodingPolicy,
FSDirErasureCodingOp#enableErasureCodingPolicy,
FSDirErasureCodingOp#disableErasureCodingPolicy functions.
{code}
> Support erasure coding policy operations in namenode edit log
> -
>
> Key: HDFS-12395
> URL: https://issues.apache.org/jira/browse/HDFS-12395
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Reporter: SammiChen
>Assignee: SammiChen
> Labels: hdfs-ec-3.0-must-do
> Attachments: editsStored, HDFS-12395.001.patch, HDFS-12395.002.patch
>
>
> Support add, remove, disable, enable erasure coding policy operation in edit
> log.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[ https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161098#comment-16161098 ] Yiqun Lin commented on HDFS-12370: -- Thanks for the review and commit, [~cheersyang].I noticed you mentioned one point that current deletion behaviour is really slow. bq. I think that was because the throttle logic at KSM or SCM side were non-optimal Now we did the throttle logic across many phases. Can we throttle it only in source place by default? I mean that we just do this in KSM since deletion operation is invoked fistly in KSM side. Since the source place is throttled, I think there will also not be much pending deletion blocks that to be deleted afterward. Users can enable throttle in SCM if they want. > Ozone: Implement TopN container choosing policy for BlockDeletionService > > > Key: HDFS-12370 > URL: https://issues.apache.org/jira/browse/HDFS-12370 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12370-HDFS-7240.001.patch, > HDFS-12370-HDFS-7240.002.patch, HDFS-12370-HDFS-7240.003.patch > > > Implement TopN container choosing policy for BlockDeletionService. This is > discussed from HDFS-12354. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[ https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161170#comment-16161170 ] Weiwei Yang commented on HDFS-12370: Hi [~linyiqun] Thanks for sharing your idea. The problem here is that different layers are using different message format to transfer deletion messages, so it is not easy to define a single throttle mechanism for all these places. You can find more details in the up-to-date [^document|https://issues.apache.org/jira/secure/attachment/12886359/Asynchronous%20key%20delete%20.pdf] I uploaded in HDFS-11922, please take a look at the document and let me know if you have any good idea to resolve this, thanks! > Ozone: Implement TopN container choosing policy for BlockDeletionService > > > Key: HDFS-12370 > URL: https://issues.apache.org/jira/browse/HDFS-12370 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12370-HDFS-7240.001.patch, > HDFS-12370-HDFS-7240.002.patch, HDFS-12370-HDFS-7240.003.patch > > > Implement TopN container choosing policy for BlockDeletionService. This is > discussed from HDFS-12354. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12414) Ensure to use CLI command to enable/disable erasure coding policy
[
https://issues.apache.org/jira/browse/HDFS-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161140#comment-16161140
]
Hadoop QA commented on HDFS-12414:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 44 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
39s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 48s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 7 new + 1116 unchanged - 1 fixed = 1123 total (was 1117) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
48s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestPread |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.hdfs.tools.TestDFSAdminWithHA |
| | hadoop.tracing.TestTracing |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 |
| | hadoop.hdfs.server.blockmanagement.TestBlockManager |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 |
| |
[jira] [Commented] (HDFS-12415) Ozone: TestXceiverClientManager occasionally fails
[
https://issues.apache.org/jira/browse/HDFS-12415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161137#comment-16161137
]
Mukul Kumar Singh commented on HDFS-12415:
--
Thanks for the patch [~cheersyang], The patch looks good. However it seems that
MiniOzoneCluster.build() already calls {{cluster.waitOzoneReady()}}
{code}
try {
cluster.waitOzoneReady();
if (waitForChillModeFinish) {
cluster.waitTobeOutOfChillMode();
}
cluster.waitForHeartbeatProcessed();
} catch (Exception e) {
{code}
> Ozone: TestXceiverClientManager occasionally fails
> --
>
> Key: HDFS-12415
> URL: https://issues.apache.org/jira/browse/HDFS-12415
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: HDFS-12415-HDFS-7240.001.patch
>
>
> TestXceiverClientManager seems to be occasionally failing in some jenkins
> jobs,
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.scm.node.SCMNodeManager.getNodeStat(SCMNodeManager.java:828)
> at
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.hasEnoughSpace(SCMCommonPolicy.java:147)
> at
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.lambda$chooseDatanodes$0(SCMCommonPolicy.java:125)
> {noformat}
> see more from [this
> report|https://builds.apache.org/job/PreCommit-HDFS-Build/21065/testReport/]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12415) Ozone: TestXceiverClientManager occasionally fails
[
https://issues.apache.org/jira/browse/HDFS-12415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161168#comment-16161168
]
Weiwei Yang commented on HDFS-12415:
Hi [~msingh]
Thanks for pointing this out, you are right, that is already called. Then I am
not sure why SCMNodeManager.getNodeStat() gets a NPE there, reopen this for
more investigation. Please feel free to comment if you have suggestion how to
get this fixed, thank you.
> Ozone: TestXceiverClientManager occasionally fails
> --
>
> Key: HDFS-12415
> URL: https://issues.apache.org/jira/browse/HDFS-12415
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: HDFS-12415-HDFS-7240.001.patch
>
>
> TestXceiverClientManager seems to be occasionally failing in some jenkins
> jobs,
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.scm.node.SCMNodeManager.getNodeStat(SCMNodeManager.java:828)
> at
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.hasEnoughSpace(SCMCommonPolicy.java:147)
> at
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.lambda$chooseDatanodes$0(SCMCommonPolicy.java:125)
> {noformat}
> see more from [this
> report|https://builds.apache.org/job/PreCommit-HDFS-Build/21065/testReport/]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12415) Ozone: TestXceiverClientManager occasionally fails
[
https://issues.apache.org/jira/browse/HDFS-12415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12415:
---
Status: In Progress (was: Patch Available)
> Ozone: TestXceiverClientManager occasionally fails
> --
>
> Key: HDFS-12415
> URL: https://issues.apache.org/jira/browse/HDFS-12415
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: HDFS-12415-HDFS-7240.001.patch
>
>
> TestXceiverClientManager seems to be occasionally failing in some jenkins
> jobs,
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.scm.node.SCMNodeManager.getNodeStat(SCMNodeManager.java:828)
> at
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.hasEnoughSpace(SCMCommonPolicy.java:147)
> at
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.lambda$chooseDatanodes$0(SCMCommonPolicy.java:125)
> {noformat}
> see more from [this
> report|https://builds.apache.org/job/PreCommit-HDFS-Build/21065/testReport/]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[ https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161170#comment-16161170 ] Weiwei Yang edited comment on HDFS-12370 at 9/11/17 12:50 PM: -- Hi [~linyiqun] Thanks for sharing your idea. The problem here is that different layers are using different message format to transfer deletion messages, so it is not easy to define a single throttle mechanism for all these places. You can find more details in the up-to-date [document|https://issues.apache.org/jira/secure/attachment/12886359/Asynchronous%20key%20delete%20.pdf] I uploaded in HDFS-11922, please take a look at the document and let me know if you have any good idea to resolve this, thanks! was (Author: cheersyang): Hi [~linyiqun] Thanks for sharing your idea. The problem here is that different layers are using different message format to transfer deletion messages, so it is not easy to define a single throttle mechanism for all these places. You can find more details in the up-to-date [^document|https://issues.apache.org/jira/secure/attachment/12886359/Asynchronous%20key%20delete%20.pdf] I uploaded in HDFS-11922, please take a look at the document and let me know if you have any good idea to resolve this, thanks! > Ozone: Implement TopN container choosing policy for BlockDeletionService > > > Key: HDFS-12370 > URL: https://issues.apache.org/jira/browse/HDFS-12370 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12370-HDFS-7240.001.patch, > HDFS-12370-HDFS-7240.002.patch, HDFS-12370-HDFS-7240.003.patch > > > Implement TopN container choosing policy for BlockDeletionService. This is > discussed from HDFS-12354. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11922) Ozone: KSM: Garbage collect deleted blocks
[ https://issues.apache.org/jira/browse/HDFS-11922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-11922: --- Attachment: Asynchronous key delete .pdf All major tasks are done, updated design doc updated, closing this task now. Thanks all. > Ozone: KSM: Garbage collect deleted blocks > -- > > Key: HDFS-11922 > URL: https://issues.apache.org/jira/browse/HDFS-11922 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Weiwei Yang >Priority: Critical > Attachments: Asynchronous key delete .pdf > > > We need to garbage collect deleted blocks from the Datanodes. There are two > cases where we will have orphaned blocks. One is like the classical HDFS, > where someone deletes a key and we need to delete the corresponding blocks. > Another case, is when someone overwrites a key -- an overwrite can be treated > as a delete and a new put -- that means that older blocks need to be GC-ed at > some point of time. > Couple of JIRAs has discussed this in one form or another -- so consolidating > all those discussions in this JIRA. > HDFS-11796 -- needs to fix this issue for some tests to pass > HDFS-11780 -- changed the old overwriting behavior to not supporting this > feature for time being. > HDFS-11920 - Once again runs into this issue when user tries to put an > existing key. > HDFS-11781 - delete key API in KSM only deletes the metadata -- and relies on > GC for Datanodes. > When we solve this issue, we should also consider 2 more aspects. > One, we support versioning in the buckets, tracking which blocks are really > orphaned is something that KSM will do. So delete and overwrite at some point > needs to decide how to handle versioning of buckets. > Two, If a key exists in a closed container, then it is immutable, hence the > strategy of removing the key might be more complex than just talking to an > open container. > cc : [~xyao], [~cheersyang], [~vagarychen], [~msingh], [~yuanbo], > [~szetszwo], [~nandakumar131] > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12282) Ozone: DeleteKey-4: Block delete between SCM and DN
[ https://issues.apache.org/jira/browse/HDFS-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12282: --- Summary: Ozone: DeleteKey-4: Block delete between SCM and DN (was: Ozone: DeleteKey-4: Block delete via HB between SCM and DN) > Ozone: DeleteKey-4: Block delete between SCM and DN > --- > > Key: HDFS-12282 > URL: https://issues.apache.org/jira/browse/HDFS-12282 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, ozone, scm >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Fix For: HDFS-7240 > > Attachments: Block delete via HB between SCM and DN.png, Block delete > via HB between SCM and DN v2.png, HDFS-12282-HDFS-7240.001.patch, > HDFS-12282-HDFS-7240.002.patch, HDFS-12282-HDFS-7240.003.patch, > HDFS-12282-HDFS-7240.004.patch, HDFS-12282-HDFS-7240.005.patch, > HDFS-12282.WIP.patch > > > This is the task 3 in the design doc, implements the SCM to datanode > interactions. Including > # SCM sends block deletion message via HB to datanode > # datanode changes block state to deleting when processes the HB response > # datanode sends deletion ACKs back to SCM > # SCM handles ACKs and removes blocks in DB -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12235) Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions
[ https://issues.apache.org/jira/browse/HDFS-12235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12235: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Status: Resolved (was: Patch Available) I've committed this to the feature branch, thanks for all the reviews [~anu], [~vagarychen] [~nandakumar131]. > Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions > --- > > Key: HDFS-12235 > URL: https://issues.apache.org/jira/browse/HDFS-12235 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12235-HDFS-7240.001.patch, > HDFS-12235-HDFS-7240.002.patch, HDFS-12235-HDFS-7240.003.patch, > HDFS-12235-HDFS-7240.004.patch, HDFS-12235-HDFS-7240.005.patch, > HDFS-12235-HDFS-7240.006.patch, HDFS-12235-HDFS-7240.007.patch, > HDFS-12235-HDFS-7240.008.patch, HDFS-12235-HDFS-7240.009.patch, > HDFS-12235-HDFS-7240.010.patch, HDFS-12235-HDFS-7240.011.patch, > HDFS-12235-HDFS-7240.012.patch > > > KSM and SCM interaction for delete key operation, both KSM and SCM stores key > state info in a backlog, KSM needs to scan this log and send block-deletion > command to SCM, once SCM is fully aware of the message, KSM removes the key > completely from namespace. See more from the design doc under HDFS-11922, > this is task break down 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11922) Ozone: KSM: Garbage collect deleted blocks
[ https://issues.apache.org/jira/browse/HDFS-11922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-11922: --- Attachment: (was: Async delete keys.pdf) > Ozone: KSM: Garbage collect deleted blocks > -- > > Key: HDFS-11922 > URL: https://issues.apache.org/jira/browse/HDFS-11922 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Weiwei Yang >Priority: Critical > > We need to garbage collect deleted blocks from the Datanodes. There are two > cases where we will have orphaned blocks. One is like the classical HDFS, > where someone deletes a key and we need to delete the corresponding blocks. > Another case, is when someone overwrites a key -- an overwrite can be treated > as a delete and a new put -- that means that older blocks need to be GC-ed at > some point of time. > Couple of JIRAs has discussed this in one form or another -- so consolidating > all those discussions in this JIRA. > HDFS-11796 -- needs to fix this issue for some tests to pass > HDFS-11780 -- changed the old overwriting behavior to not supporting this > feature for time being. > HDFS-11920 - Once again runs into this issue when user tries to put an > existing key. > HDFS-11781 - delete key API in KSM only deletes the metadata -- and relies on > GC for Datanodes. > When we solve this issue, we should also consider 2 more aspects. > One, we support versioning in the buckets, tracking which blocks are really > orphaned is something that KSM will do. So delete and overwrite at some point > needs to decide how to handle versioning of buckets. > Two, If a key exists in a closed container, then it is immutable, hence the > strategy of removing the key might be more complex than just talking to an > open container. > cc : [~xyao], [~cheersyang], [~vagarychen], [~msingh], [~yuanbo], > [~szetszwo], [~nandakumar131] > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-11922) Ozone: KSM: Garbage collect deleted blocks
[ https://issues.apache.org/jira/browse/HDFS-11922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang resolved HDFS-11922. Resolution: Done Fix Version/s: HDFS-7240 Target Version/s: HDFS-7240 > Ozone: KSM: Garbage collect deleted blocks > -- > > Key: HDFS-11922 > URL: https://issues.apache.org/jira/browse/HDFS-11922 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Weiwei Yang >Priority: Critical > Fix For: HDFS-7240 > > Attachments: Asynchronous key delete .pdf > > > We need to garbage collect deleted blocks from the Datanodes. There are two > cases where we will have orphaned blocks. One is like the classical HDFS, > where someone deletes a key and we need to delete the corresponding blocks. > Another case, is when someone overwrites a key -- an overwrite can be treated > as a delete and a new put -- that means that older blocks need to be GC-ed at > some point of time. > Couple of JIRAs has discussed this in one form or another -- so consolidating > all those discussions in this JIRA. > HDFS-11796 -- needs to fix this issue for some tests to pass > HDFS-11780 -- changed the old overwriting behavior to not supporting this > feature for time being. > HDFS-11920 - Once again runs into this issue when user tries to put an > existing key. > HDFS-11781 - delete key API in KSM only deletes the metadata -- and relies on > GC for Datanodes. > When we solve this issue, we should also consider 2 more aspects. > One, we support versioning in the buckets, tracking which blocks are really > orphaned is something that KSM will do. So delete and overwrite at some point > needs to decide how to handle versioning of buckets. > Two, If a key exists in a closed container, then it is immutable, hence the > strategy of removing the key might be more complex than just talking to an > open container. > cc : [~xyao], [~cheersyang], [~vagarychen], [~msingh], [~yuanbo], > [~szetszwo], [~nandakumar131] > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11922) Ozone: KSM: Garbage collect deleted blocks
[ https://issues.apache.org/jira/browse/HDFS-11922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161409#comment-16161409 ] Anu Engineer commented on HDFS-11922: - [~cheersyang] & [~yuanbo] Thanks for designing and getting this work done. It is amazing and takes us very close to the ozone trunk merge. Really appreciate all your hard work. Thanks to all the reviewers too, really appreciate all of you pitching in to complete this work. > Ozone: KSM: Garbage collect deleted blocks > -- > > Key: HDFS-11922 > URL: https://issues.apache.org/jira/browse/HDFS-11922 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Weiwei Yang >Priority: Critical > Fix For: HDFS-7240 > > Attachments: Asynchronous key delete .pdf > > > We need to garbage collect deleted blocks from the Datanodes. There are two > cases where we will have orphaned blocks. One is like the classical HDFS, > where someone deletes a key and we need to delete the corresponding blocks. > Another case, is when someone overwrites a key -- an overwrite can be treated > as a delete and a new put -- that means that older blocks need to be GC-ed at > some point of time. > Couple of JIRAs has discussed this in one form or another -- so consolidating > all those discussions in this JIRA. > HDFS-11796 -- needs to fix this issue for some tests to pass > HDFS-11780 -- changed the old overwriting behavior to not supporting this > feature for time being. > HDFS-11920 - Once again runs into this issue when user tries to put an > existing key. > HDFS-11781 - delete key API in KSM only deletes the metadata -- and relies on > GC for Datanodes. > When we solve this issue, we should also consider 2 more aspects. > One, we support versioning in the buckets, tracking which blocks are really > orphaned is something that KSM will do. So delete and overwrite at some point > needs to decide how to handle versioning of buckets. > Two, If a key exists in a closed container, then it is immutable, hence the > strategy of removing the key might be more complex than just talking to an > open container. > cc : [~xyao], [~cheersyang], [~vagarychen], [~msingh], [~yuanbo], > [~szetszwo], [~nandakumar131] > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11754) Make FsServerDefaults cache configurable.
[
https://issues.apache.org/jira/browse/HDFS-11754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161415#comment-16161415
]
Rushabh S Shah commented on HDFS-11754:
---
Thanks [~erofeev] for working on this.
Overall the patch looks good.
I have one concern in unit test.
Already the hdfs unit tests runs for a long time. We should minimize the number
of times the namenode needs to be bounced.
The only reason we are bouncing the namenode to change the config value of
{{DFS_BLOCK_SIZE_DEFAULT}}
You can use {{NameNodeAdapter#spyOnNamesystem()}} and return whatever
{{FsServerDefaults}} object you want. That way you don't need to bounce
namenode.
Hope that makes sense.
One minor nit.
1. HdfsClientConfigKeys.java
Rename the config key name from
{{dfs.client.server.defaults.validity.period.ms}} to
{{dfs.client.server-defaults.validity.period.ms}}
Regarding test failures, there are lot of EC related tests that are failing
consistently.
Just cross check running all the tests with and without your patch to verify
that none of the tests failed because of your patch.
Regarding 1 ASF License warnings, if you believe that your patch didn't cause
that warning then please open a jira and hope someone from build team will take
a look.
Regarding the findbugs warning, I believe that your patch didn't cause that
warning. Just verify once that my understanding is correct.
> Make FsServerDefaults cache configurable.
> -
>
> Key: HDFS-11754
> URL: https://issues.apache.org/jira/browse/HDFS-11754
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Mikhail Erofeev
>Priority: Minor
> Labels: newbie
> Fix For: 2.9.0
>
> Attachments: HDFS-11754.001.patch, HDFS-11754.002.patch,
> HDFS-11754.003.patch, HDFS-11754.004.patch
>
>
> DFSClient caches the result of FsServerDefaults for 60 minutes.
> But the 60 minutes time is not configurable.
> Continuing the discussion from HDFS-11702, it would be nice if we can make
> this configurable and make the default as 60 minutes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12235) Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions
[ https://issues.apache.org/jira/browse/HDFS-12235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160790#comment-16160790 ] Weiwei Yang commented on HDFS-12235: The UT failure was not related, I will commit this shortly. > Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions > --- > > Key: HDFS-12235 > URL: https://issues.apache.org/jira/browse/HDFS-12235 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Labels: ozoneMerge > Attachments: HDFS-12235-HDFS-7240.001.patch, > HDFS-12235-HDFS-7240.002.patch, HDFS-12235-HDFS-7240.003.patch, > HDFS-12235-HDFS-7240.004.patch, HDFS-12235-HDFS-7240.005.patch, > HDFS-12235-HDFS-7240.006.patch, HDFS-12235-HDFS-7240.007.patch, > HDFS-12235-HDFS-7240.008.patch, HDFS-12235-HDFS-7240.009.patch, > HDFS-12235-HDFS-7240.010.patch, HDFS-12235-HDFS-7240.011.patch, > HDFS-12235-HDFS-7240.012.patch > > > KSM and SCM interaction for delete key operation, both KSM and SCM stores key > state info in a backlog, KSM needs to scan this log and send block-deletion > command to SCM, once SCM is fully aware of the message, KSM removes the key > completely from namespace. See more from the design doc under HDFS-11922, > this is task break down 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11600) Refactor TestDFSStripedOutputStreamWithFailure test classes
[
https://issues.apache.org/jira/browse/HDFS-11600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HDFS-11600:
-
Attachment: HDFS-11600.002.patch
> Refactor TestDFSStripedOutputStreamWithFailure test classes
> ---
>
> Key: HDFS-11600
> URL: https://issues.apache.org/jira/browse/HDFS-11600
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Affects Versions: 3.0.0-alpha2
>Reporter: Andrew Wang
>Assignee: Elek, Marton
>Priority: Minor
> Attachments: HDFS-11600.002.patch, HDFS-11600-1.patch
>
>
> TestDFSStripedOutputStreamWithFailure has a great number of subclasses. The
> tests are parameterized based on the name of these subclasses.
> Seems like we could parameterize these tests with JUnit and then not need all
> these separate test classes.
> Another note, the tests will randomly return instead of running the test.
> Using {{Assume}} instead would make it more clear in the test output that
> these tests were skipped.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11600) Refactor TestDFSStripedOutputStreamWithFailure test classes
[
https://issues.apache.org/jira/browse/HDFS-11600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161842#comment-16161842
]
Chris Douglas commented on HDFS-11600:
--
Attached a naive rebase of the original patch. [~arpitagarwal], could you take
a look?
> Refactor TestDFSStripedOutputStreamWithFailure test classes
> ---
>
> Key: HDFS-11600
> URL: https://issues.apache.org/jira/browse/HDFS-11600
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Affects Versions: 3.0.0-alpha2
>Reporter: Andrew Wang
>Assignee: Elek, Marton
>Priority: Minor
> Attachments: HDFS-11600.002.patch, HDFS-11600-1.patch
>
>
> TestDFSStripedOutputStreamWithFailure has a great number of subclasses. The
> tests are parameterized based on the name of these subclasses.
> Seems like we could parameterize these tests with JUnit and then not need all
> these separate test classes.
> Another note, the tests will randomly return instead of running the test.
> Using {{Assume}} instead would make it more clear in the test output that
> these tests were skipped.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12417) Disable flaky TestDFSStripedOutputStreamWithFailure
[
https://issues.apache.org/jira/browse/HDFS-12417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161796#comment-16161796
]
Hadoop QA commented on HDFS-12417:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
46s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| | hadoop.hdfs.server.blockmanagement.TestBlockManager |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer |
| | hadoop.hdfs.server.namenode.TestFSImage |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12417 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886453/HDFS-12417.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d42b2982125f 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
[jira] [Commented] (HDFS-12410) Ignore unknown StorageTypes
[ https://issues.apache.org/jira/browse/HDFS-12410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161798#comment-16161798 ] Ajay Kumar commented on HDFS-12410: --- [~chris.douglas] Instead of ignoring such storage types shall we assign them a default storage type. In a scenario where end user mis-spelled storage type, first case(i.e ignoring them) will result in disks being not used for storage. However if we assign all unknown storage type to some default type and log an error message, end user can always go back and correct it. > Ignore unknown StorageTypes > --- > > Key: HDFS-12410 > URL: https://issues.apache.org/jira/browse/HDFS-12410 > Project: Hadoop HDFS > Issue Type: Task > Components: datanode, fs >Reporter: Chris Douglas >Assignee: Ajay Kumar >Priority: Minor > > A storage configured with an unknown type will cause runtime exceptions. > Instead, these storages can be ignored/skipped. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12410) Ignore unknown StorageTypes
[
https://issues.apache.org/jira/browse/HDFS-12410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161810#comment-16161810
]
Íñigo Goiri commented on HDFS-12410:
[~ajayydv] resorting to default might be another option.
However, in our original use case, the problem was using the same
{{hdfs-site.xml}} for a version of Hadoop that supported {{PROVIDED}} and
another without it.
With the current behavior the Datanode doesn't start at all.
With your approach, I'm not sure what would try to do with the {{PROVIDED}}
file path if it assumes {{DEFAULT}}.
For our use case, I'd prefer to log and ignore.
> Ignore unknown StorageTypes
> ---
>
> Key: HDFS-12410
> URL: https://issues.apache.org/jira/browse/HDFS-12410
> Project: Hadoop HDFS
> Issue Type: Task
> Components: datanode, fs
>Reporter: Chris Douglas
>Assignee: Ajay Kumar
>Priority: Minor
>
> A storage configured with an unknown type will cause runtime exceptions.
> Instead, these storages can be ignored/skipped.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12417) Disable flaky TestDFSStripedOutputStreamWithFailure
[ https://issues.apache.org/jira/browse/HDFS-12417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HDFS-12417: - Attachment: HDFS-12417.001.patch Parameterizing tests by subclassing is not the correct pattern. Rebased HDFS-11600. Will commit this after it goes in. > Disable flaky TestDFSStripedOutputStreamWithFailure > --- > > Key: HDFS-12417 > URL: https://issues.apache.org/jira/browse/HDFS-12417 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Chris Douglas > Attachments: HDFS-12417.000.patch, HDFS-12417.001.patch > > > Some subset of TestDFSStripedOutputStreamWithFailure tests almost always fail > in test-patch runs. Since its failure is no longer seen as a blocker for > commit, it should be disabled until it is more reliable. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
