[jira] [Commented] (HDFS-11804) KMS client needs retry logic
[ https://issues.apache.org/jira/browse/HDFS-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045409#comment-16045409 ] John Zhuge commented on HDFS-11804: --- [~shahrs87] Should this JIRA be moved to project "Hadoop Common" since KMS belongs to common? > KMS client needs retry logic > > > Key: HDFS-11804 > URL: https://issues.apache.org/jira/browse/HDFS-11804 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, > HDFS-11804-trunk-3.patch, HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, > HDFS-11804-trunk.patch > > > The kms client appears to have no retry logic – at all. It's completely > decoupled from the ipc retry logic. This has major impacts if the KMS is > unreachable for any reason, including but not limited to network connection > issues, timeouts, the +restart during an upgrade+. > This has some major ramifications: > # Jobs may fail to submit, although oozie resubmit logic should mask it > # Non-oozie launchers may experience higher rates if they do not already have > retry logic. > # Tasks reading EZ files will fail, probably be masked by framework reattempts > # EZ file creation fails after creating a 0-length file – client receives > EDEK in the create response, then fails when decrypting the EDEK > # Bulk hadoop fs copies, and maybe distcp, will prematurely fail -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11670) [SPS]: Add CLI command for satisfy storage policy operations
[
https://issues.apache.org/jira/browse/HDFS-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045387#comment-16045387
]
Rakesh R commented on HDFS-11670:
-
FYI, I've updated the jira subject line & description to reflect the proposed
changes.
> [SPS]: Add CLI command for satisfy storage policy operations
>
>
> Key: HDFS-11670
> URL: https://issues.apache.org/jira/browse/HDFS-11670
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-11670-HDFS-10285.001.patch
>
>
> This jira to discuss and implement set of satisfy storage policy
> sub-commands. Following are the list of sub-commands:
> # Schedule blocks to move based on file/directory policy:
> {code}hdfs storagepolicies -satisfyStoragePolicy -path ]{code}
> # Its good to have one command to check SPS is enabled or not. Based on this
> user can take the decision to run the Mover:
> {code}
> hdfs storagepolicies -isSPSRunning
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11670) [SPS]: Add CLI command for satisfy storage policy operations
[
https://issues.apache.org/jira/browse/HDFS-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-11670:
Description:
This jira to discuss and implement set of satisfy storage policy sub-commands.
Following are the list of sub-commands:
# Schedule blocks to move based on file/directory policy:
{code}hdfs storagepolicies -satisfyStoragePolicy -path ]{code}
# Its good to have one command to check SPS is enabled or not. Based on this
user can take the decision to run the Mover:
{code}
hdfs storagepolicies -isSPSRunning
{code}
was:
Its good to have one command to check SPS is enabled or not. Based on this user
can take the decision to run the Mover.
{noformat}
hdfs storagepolicies -policysatisfierstatus
{noformat}
Summary: [SPS]: Add CLI command for satisfy storage policy operations
(was: [SPS]: Add storagepolicy command to check the status of storage policy
Satisfier)
> [SPS]: Add CLI command for satisfy storage policy operations
>
>
> Key: HDFS-11670
> URL: https://issues.apache.org/jira/browse/HDFS-11670
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-11670-HDFS-10285.001.patch
>
>
> This jira to discuss and implement set of satisfy storage policy
> sub-commands. Following are the list of sub-commands:
> # Schedule blocks to move based on file/directory policy:
> {code}hdfs storagepolicies -satisfyStoragePolicy -path ]{code}
> # Its good to have one command to check SPS is enabled or not. Based on this
> user can take the decision to run the Mover:
> {code}
> hdfs storagepolicies -isSPSRunning
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11782) Ozone: KSM: Add listKey
[ https://issues.apache.org/jira/browse/HDFS-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11782: - Status: Patch Available (was: Open) > Ozone: KSM: Add listKey > --- > > Key: HDFS-11782 > URL: https://issues.apache.org/jira/browse/HDFS-11782 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: HDFS-7240 >Affects Versions: ozone >Reporter: Anu Engineer >Assignee: Yiqun Lin > Attachments: HDFS-11782-HDFS-7240.001.patch > > > Add support for listing keys in a bucket. Just like other 2 list operations, > this API supports paging via, prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11782) Ozone: KSM: Add listKey
[ https://issues.apache.org/jira/browse/HDFS-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11782: - Attachment: HDFS-11782-HDFS-7240.001.patch Attach the initial patch. Since HDFS-11926 has already done some work in getting a range of keys from levelDB, the logic of this patch is very similar to list buckets in KSM(HDFS-11779) . Kindly review. > Ozone: KSM: Add listKey > --- > > Key: HDFS-11782 > URL: https://issues.apache.org/jira/browse/HDFS-11782 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: HDFS-7240 >Affects Versions: ozone >Reporter: Anu Engineer >Assignee: Yiqun Lin > Attachments: HDFS-11782-HDFS-7240.001.patch > > > Add support for listing keys in a bucket. Just like other 2 list operations, > this API supports paging via, prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11964) Fix TestDFSStripedInputStreamWithRandomECPolicy#testPreadWithDNFailure failure
[
https://issues.apache.org/jira/browse/HDFS-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-11964:
-
Component/s: erasure-coding
> Fix TestDFSStripedInputStreamWithRandomECPolicy#testPreadWithDNFailure failure
> --
>
> Key: HDFS-11964
> URL: https://issues.apache.org/jira/browse/HDFS-11964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>
> TestDFSStripedInputStreamWithRandomECPolicy#testPreadWithDNFailure fails on
> trunk:
> {code}
> Running org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.99 sec <<<
> FAILURE! - in
> org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
> testPreadWithDNFailure(org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy)
> Time elapsed: 1.265 sec <<< FAILURE!
> org.junit.internal.ArrayComparisonFailure: arrays first differed at element
> [327680]; expected:<-36> but was:<2>
> at
> org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50)
> at org.junit.Assert.internalArrayEquals(Assert.java:473)
> at org.junit.Assert.assertArrayEquals(Assert.java:294)
> at org.junit.Assert.assertArrayEquals(Assert.java:305)
> at
> org.apache.hadoop.hdfs.TestDFSStripedInputStream.testPreadWithDNFailure(TestDFSStripedInputStream.java:306)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Moved] (HDFS-11964) Fix TestDFSStripedInputStreamWithRandomECPolicy#testPreadWithDNFailure failure
[
https://issues.apache.org/jira/browse/HDFS-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu moved HADOOP-14517 to HDFS-11964:
---
Affects Version/s: (was: 3.0.0-alpha3)
3.0.0-alpha3
Target Version/s: 3.0.0-alpha4 (was: 3.0.0-alpha4)
Key: HDFS-11964 (was: HADOOP-14517)
Project: Hadoop HDFS (was: Hadoop Common)
> Fix TestDFSStripedInputStreamWithRandomECPolicy#testPreadWithDNFailure failure
> --
>
> Key: HDFS-11964
> URL: https://issues.apache.org/jira/browse/HDFS-11964
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>
> TestDFSStripedInputStreamWithRandomECPolicy#testPreadWithDNFailure fails on
> trunk:
> {code}
> Running org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.99 sec <<<
> FAILURE! - in
> org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
> testPreadWithDNFailure(org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy)
> Time elapsed: 1.265 sec <<< FAILURE!
> org.junit.internal.ArrayComparisonFailure: arrays first differed at element
> [327680]; expected:<-36> but was:<2>
> at
> org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50)
> at org.junit.Assert.internalArrayEquals(Assert.java:473)
> at org.junit.Assert.assertArrayEquals(Assert.java:294)
> at org.junit.Assert.assertArrayEquals(Assert.java:305)
> at
> org.apache.hadoop.hdfs.TestDFSStripedInputStream.testPreadWithDNFailure(TestDFSStripedInputStream.java:306)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-11962) Ozone: Add stop-ozone.sh script
[ https://issues.apache.org/jira/browse/HDFS-11962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-11962: --- Comment: was deleted (was: I can work on this, just one question, do we need to add ozone start/stop in start-all/stop-all scripts? Probably yes?) > Ozone: Add stop-ozone.sh script > --- > > Key: HDFS-11962 > URL: https://issues.apache.org/jira/browse/HDFS-11962 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Weiwei Yang > > This script should stop the cluster along with all ozone services. -- > # run 'stop-dfs.sh' > # run 'hdfs --daemon stop ksm' > # run 'hdfs --daemon stop scm`' -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11962) Ozone: Add stop-ozone.sh script
[ https://issues.apache.org/jira/browse/HDFS-11962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045327#comment-16045327 ] Weiwei Yang commented on HDFS-11962: I can work on this, just one question, do we need to add ozone start/stop in start-all/stop-all scripts? Probably yes? > Ozone: Add stop-ozone.sh script > --- > > Key: HDFS-11962 > URL: https://issues.apache.org/jira/browse/HDFS-11962 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Weiwei Yang > > This script should stop the cluster along with all ozone services. -- > # run 'stop-dfs.sh' > # run 'hdfs --daemon stop ksm' > # run 'hdfs --daemon stop scm`' -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11961) Ozone: Add start-ozone.sh to quickly start ozone.
[ https://issues.apache.org/jira/browse/HDFS-11961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang reassigned HDFS-11961: -- Assignee: Weiwei Yang > Ozone: Add start-ozone.sh to quickly start ozone. > - > > Key: HDFS-11961 > URL: https://issues.apache.org/jira/browse/HDFS-11961 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Weiwei Yang > > Add start ozone script. Internally this script should call into > # start-dfs.sh > # run `hdfs --daemon start scm' > # run `hdfs --daemon start ksm` > This just makes it easy to start Ozone with a single command. This command > assumes that Namenode format has been run before, since it will bring up HDFS > also. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11962) Ozone: Add stop-ozone.sh script
[ https://issues.apache.org/jira/browse/HDFS-11962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang reassigned HDFS-11962: -- Assignee: Weiwei Yang > Ozone: Add stop-ozone.sh script > --- > > Key: HDFS-11962 > URL: https://issues.apache.org/jira/browse/HDFS-11962 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Weiwei Yang > > This script should stop the cluster along with all ozone services. -- > # run 'stop-dfs.sh' > # run 'hdfs --daemon stop ksm' > # run 'hdfs --daemon stop scm`' -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11939) Ozone : add read/write random access to Chunks of a key
[
https://issues.apache.org/jira/browse/HDFS-11939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045302#comment-16045302
]
Hadoop QA commented on HDFS-11939:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
50s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
4s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
57s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 1 new
+ 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
25s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}122m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
34s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
| | Inconsistent synchronization of
org.apache.hadoop.scm.storage.ChunkOutputChannel.buffer; locked 64% of time
Unsynchronized access at ChunkOutputChannel.java:64% of time Unsynchronized
access at ChunkOutputChannel.java:[line 230] |
| Failed junit tests | hadoop.hdfs.TestMaintenanceState |
| | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.ozone.container.ozoneimpl.TestOzoneContainer |
| | hadoop.cblock.TestBufferManager |
| | hadoop.ozone.scm.node.TestNodeManager |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | org.apache.hadoop.cblock.TestLocalBlockCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11939 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872363/HDFS-11939-HDFS-7240.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 37083ef39f4c 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11
16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Updated] (HDFS-11882) Client fails if acknowledged size is greater than bytes sent
[
https://issues.apache.org/jira/browse/HDFS-11882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-11882:
---
Labels: hdfs-ec-3.0-nice-to-have (was: )
Priority: Critical (was: Major)
I'm bumping the priority of this based on my understanding of the bug. Not sure
if it's quite a blocker, but it needs to be looked into more deeply.
> Client fails if acknowledged size is greater than bytes sent
>
>
> Key: HDFS-11882
> URL: https://issues.apache.org/jira/browse/HDFS-11882
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Critical
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-11882.01.patch
>
>
> Some tests of erasure coding fails by the following exception. The following
> test was removed by HDFS-11823, however, this type of error can happen in
> real cluster.
> {noformat}
> Running
> org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure
> Tests run: 14, Failures: 0, Errors: 1, Skipped: 10, Time elapsed: 89.086 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure
> testMultipleDatanodeFailure56(org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure)
> Time elapsed: 38.831 sec <<< ERROR!
> java.lang.IllegalStateException: null
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.updatePipeline(DFSStripedOutputStream.java:780)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.checkStreamerFailures(DFSStripedOutputStream.java:664)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.closeImpl(DFSStripedOutputStream.java:1034)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:842)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTest(TestDFSStripedOutputStreamWithFailure.java:472)
> at
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTestWithMultipleFailure(TestDFSStripedOutputStreamWithFailure.java:381)
> at
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.testMultipleDatanodeFailure56(TestDFSStripedOutputStreamWithFailure.java:245)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11907) Add metric for time taken by NameNode resource check
[
https://issues.apache.org/jira/browse/HDFS-11907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045256#comment-16045256
]
Hadoop QA commented on HDFS-11907:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 249 unchanged - 0 fixed = 250 total (was 249) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 42s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11907 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872355/HDFS-11907.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 04b8bf714a5d 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11
16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5578af8 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/19857/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19857/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19857/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19857/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add metric for time taken by NameNode resource check
>
[jira] [Assigned] (HDFS-11963) Ozone: Documentation: Add getting started page
[ https://issues.apache.org/jira/browse/HDFS-11963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer reassigned HDFS-11963: --- Assignee: Anu Engineer > Ozone: Documentation: Add getting started page > -- > > Key: HDFS-11963 > URL: https://issues.apache.org/jira/browse/HDFS-11963 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Anu Engineer > > We need to add the Ozone section to hadoop documentation and also a section > on how to get started, that is what are configuration settings needed to > start running ozone. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11963) Ozone: Documentation: Add getting started page
Anu Engineer created HDFS-11963: --- Summary: Ozone: Documentation: Add getting started page Key: HDFS-11963 URL: https://issues.apache.org/jira/browse/HDFS-11963 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Anu Engineer We need to add the Ozone section to hadoop documentation and also a section on how to get started, that is what are configuration settings needed to start running ozone. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11957) Enable POSIX ACL inheritance by default
[ https://issues.apache.org/jira/browse/HDFS-11957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-11957: -- Status: In Progress (was: Patch Available) > Enable POSIX ACL inheritance by default > --- > > Key: HDFS-11957 > URL: https://issues.apache.org/jira/browse/HDFS-11957 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0-alpha2 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HDFS-11957.001.patch > > > It is time to enable POSIX ACL inheritance by default. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11939) Ozone : add read/write random access to Chunks of a key
[
https://issues.apache.org/jira/browse/HDFS-11939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045179#comment-16045179
]
Hadoop QA commented on HDFS-11939:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
45s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 1 new
+ 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
22s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 28s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
22s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}108m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
| | Inconsistent synchronization of
org.apache.hadoop.scm.storage.ChunkOutputChannel.buffer; locked 64% of time
Unsynchronized access at ChunkOutputChannel.java:64% of time Unsynchronized
access at ChunkOutputChannel.java:[line 230] |
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| Timed out junit tests |
org.apache.hadoop.ozone.container.ozoneimpl.TestRatisManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11939 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872344/HDFS-11939-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 44584cd06fd9 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 0a05da9 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
|
[jira] [Comment Edited] (HDFS-11682) TestBalancer#testBalancerWithStripedFile is flaky
[
https://issues.apache.org/jira/browse/HDFS-11682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045163#comment-16045163
]
Manoj Govindassamy edited comment on HDFS-11682 at 6/9/17 10:48 PM:
[~eddyxu],
>From your explanation it looks like it is inevitable to run {{runBalancer}} a
>couple of times with updated HB to trigger additional balancing if needed. +1
>(non-binding) with one comment below.
{noformat}
942 while (retry > 0) {
943 // start rebalancing
944 Collection namenodes = DFSUtil.getInternalNsRpcUris(conf);
945 final int run = runBalancer(namenodes, p, conf);
.. ..
955 waitForHeartBeat(totalUsedSpace, totalCapacity, client, cluster);
956 LOG.info(" .");
957 try {
958 waitForBalancer(totalUsedSpace, totalCapacity, client, cluster,
p,
959 excludedNodes);
960 } catch (TimeoutException e) {
961 // See HDFS-11682. NN may not get heartbeat to reflect the
newest
962 // block changes.
963 retry--;
964 if (retry == 0) {
965 throw e;
966 }
967 LOG.warn("The cluster has not balanced yet, retry...");
968 continue;
969 }
970 break;
971 }
{noformat}
{{waitForHeartBeat}} in the above loop can also timeout and throw
{{TimeoutException}} which is not caught like in {{waitForBalancer}}. So, the
caller could fail because of this.
was (Author: manojg):
[~eddyxu],
>From your explanation it looks like it is inevitable to run {{runBalancer}} a
>couple of times with updated HB to trigger additional balancing if needed. +1
>(non-binding) with one comment below.
{noformat}
942 while (retry > 0) {
943 // start rebalancing
944 Collection namenodes = DFSUtil.getInternalNsRpcUris(conf);
945 final int run = runBalancer(namenodes, p, conf);
.. ..
955 waitForHeartBeat(totalUsedSpace, totalCapacity, client, cluster);
956 LOG.info(" .");
957 try {
958 waitForBalancer(totalUsedSpace, totalCapacity, client, cluster,
p,
959 excludedNodes);
960 } catch (TimeoutException e) {
961 // See HDFS-11682. NN may not get heartbeat to reflect the
newest
962 // block changes.
963 retry--;
964 if (retry == 0) {
965 throw e;
966 }
967 LOG.warn("The cluster has not balanced yet, retry...");
968 continue;
969 }
970 break;
971 }
{noformat}
{{waitForHeartBeat}} in the above loop can also timeout and throw
{{TimeoutException}} which is not caught like in {{waitForBalancer}}. So, the
caller could fail because of this.
> TestBalancer#testBalancerWithStripedFile is flaky
> -
>
> Key: HDFS-11682
> URL: https://issues.apache.org/jira/browse/HDFS-11682
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-11682.00.patch, HDFS-11682.01.patch,
> IndexOutOfBoundsException.log, timeout.log
>
>
> Saw this fail in two different ways on a precommit run, but pass locally.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11682) TestBalancer#testBalancerWithStripedFile is flaky
[
https://issues.apache.org/jira/browse/HDFS-11682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045163#comment-16045163
]
Manoj Govindassamy edited comment on HDFS-11682 at 6/9/17 10:48 PM:
[~eddyxu],
>From your explanation it looks like it is inevitable to run {{runBalancer}} a
>couple of times with updated HB to trigger additional balancing if needed. +1
>(non-binding) with one comment below.
{noformat}
942 while (retry > 0) {
943 // start rebalancing
944 Collection namenodes = DFSUtil.getInternalNsRpcUris(conf);
945 final int run = runBalancer(namenodes, p, conf);
.. ..
955 waitForHeartBeat(totalUsedSpace, totalCapacity, client, cluster);
956 LOG.info(" .");
957 try {
958 waitForBalancer(totalUsedSpace, totalCapacity, client, cluster,
p,
959 excludedNodes);
960 } catch (TimeoutException e) {
961 // See HDFS-11682. NN may not get heartbeat to reflect the
newest
962 // block changes.
963 retry--;
964 if (retry == 0) {
965 throw e;
966 }
967 LOG.warn("The cluster has not balanced yet, retry...");
968 continue;
969 }
970 break;
971 }
{noformat}
{{waitForHeartBeat}} in the above loop can also timeout and throw
{{TimeoutException}} which is not caught like in {{waitForBalancer}}. So, the
caller could fail because of this.
was (Author: manojg):
[~eddyxu],
>From your explanation it looks like it is inevitable to run {{runBalancer}} a
>couple of times with updated HB to trigger additional balancing if needed. +1
>(non-binding) with one comment below.
{noformat}
942 while (retry > 0) {
943 // start rebalancing
944 Collection namenodes = DFSUtil.getInternalNsRpcUris(conf);
945 final int run = runBalancer(namenodes, p, conf);
.. ..
955 waitForHeartBeat(totalUsedSpace, totalCapacity, client, cluster);
956 LOG.info(" .");
957 try {
958 waitForBalancer(totalUsedSpace, totalCapacity, client, cluster,
p,
959 excludedNodes);
960 } catch (TimeoutException e) {
961 // See HDFS-11682. NN may not get heartbeat to reflect the
newest
962 // block changes.
963 retry--;
964 if (retry == 0) {
965 throw e;
966 }
967 LOG.warn("The cluster has not balanced yet, retry...");
968 continue;
969 }
970 break;
971 }
{{waitForHeartBeat}} in the above loop can also timeout and throw
{{TimeoutException}} which is not caught like in {{waitForBalancer}}. So, the
caller could fail because of this.
> TestBalancer#testBalancerWithStripedFile is flaky
> -
>
> Key: HDFS-11682
> URL: https://issues.apache.org/jira/browse/HDFS-11682
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-11682.00.patch, HDFS-11682.01.patch,
> IndexOutOfBoundsException.log, timeout.log
>
>
> Saw this fail in two different ways on a precommit run, but pass locally.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11682) TestBalancer#testBalancerWithStripedFile is flaky
[
https://issues.apache.org/jira/browse/HDFS-11682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045163#comment-16045163
]
Manoj Govindassamy commented on HDFS-11682:
---
[~eddyxu],
>From your explanation it looks like it is inevitable to run {{runBalancer}} a
>couple of times with updated HB to trigger additional balancing if needed. +1
>(non-binding) with one comment below.
{noformat}
942 while (retry > 0) {
943 // start rebalancing
944 Collection namenodes = DFSUtil.getInternalNsRpcUris(conf);
945 final int run = runBalancer(namenodes, p, conf);
.. ..
955 waitForHeartBeat(totalUsedSpace, totalCapacity, client, cluster);
956 LOG.info(" .");
957 try {
958 waitForBalancer(totalUsedSpace, totalCapacity, client, cluster,
p,
959 excludedNodes);
960 } catch (TimeoutException e) {
961 // See HDFS-11682. NN may not get heartbeat to reflect the
newest
962 // block changes.
963 retry--;
964 if (retry == 0) {
965 throw e;
966 }
967 LOG.warn("The cluster has not balanced yet, retry...");
968 continue;
969 }
970 break;
971 }
{{waitForHeartBeat}} in the above loop can also timeout and throw
{{TimeoutException}} which is not caught like in {{waitForBalancer}}. So, the
caller could fail because of this.
> TestBalancer#testBalancerWithStripedFile is flaky
> -
>
> Key: HDFS-11682
> URL: https://issues.apache.org/jira/browse/HDFS-11682
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-11682.00.patch, HDFS-11682.01.patch,
> IndexOutOfBoundsException.log, timeout.log
>
>
> Saw this fail in two different ways on a precommit run, but pass locally.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11962) Ozone: Add stop-ozone.sh script
[ https://issues.apache.org/jira/browse/HDFS-11962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11962: Description: This script should stop the cluster along with all ozone services. -- # run 'stop-dfs.sh' # run 'hdfs --daemon stop ksm' # run 'hdfs --daemon stop scm`' was: This script should stop the cluster along with all ozone services. -- #run 'stop-dfs.sh' # run 'hdfs --daemon stop ksm' # run 'hdfs --daemon stop scm`' > Ozone: Add stop-ozone.sh script > --- > > Key: HDFS-11962 > URL: https://issues.apache.org/jira/browse/HDFS-11962 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer > > This script should stop the cluster along with all ozone services. -- > # run 'stop-dfs.sh' > # run 'hdfs --daemon stop ksm' > # run 'hdfs --daemon stop scm`' -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11962) Ozone: Add stop-ozone.sh script
Anu Engineer created HDFS-11962: --- Summary: Ozone: Add stop-ozone.sh script Key: HDFS-11962 URL: https://issues.apache.org/jira/browse/HDFS-11962 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Anu Engineer This script should stop the cluster along with all ozone services. -- #run 'stop-dfs.sh' # run 'hdfs --daemon stop ksm' # run 'hdfs --daemon stop scm`' -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11961) Ozone: Add start-ozone.sh to quickly start ozone.
Anu Engineer created HDFS-11961: --- Summary: Ozone: Add start-ozone.sh to quickly start ozone. Key: HDFS-11961 URL: https://issues.apache.org/jira/browse/HDFS-11961 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Anu Engineer Add start ozone script. Internally this script should call into # start-dfs.sh # run `hdfs --daemon start scm' # run `hdfs --daemon start ksm` This just makes it easy to start Ozone with a single command. This command assumes that Namenode format has been run before, since it will bring up HDFS also. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11939) Ozone : add read/write random access to Chunks of a key
[ https://issues.apache.org/jira/browse/HDFS-11939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11939: -- Attachment: HDFS-11939-HDFS-7240.003.patch fixing a bug in v003 patch. > Ozone : add read/write random access to Chunks of a key > --- > > Key: HDFS-11939 > URL: https://issues.apache.org/jira/browse/HDFS-11939 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11939-HDFS-7240.001.patch, > HDFS-11939-HDFS-7240.002.patch, HDFS-11939-HDFS-7240.003.patch > > > In Ozone, the value of a key is a sequence of container chunks. Currently, > the only way to read/write the chunks is by using ChunkInputStream and > ChunkOutputStream. However, by the nature of streams, these classes are > currently implemented to only allow sequential read/write. > Ideally we would like to support random access of the chunks. For example, we > want to be able to seek to a specific offset and read/write some data. This > will be critical for key range read/write feature, and potentially important > for supporting parallel read/write. > This JIRA tracks adding support by implementing FileChannel class on top > Chunks. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11907) Add metric for time taken by NameNode resource check
[
https://issues.apache.org/jira/browse/HDFS-11907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045121#comment-16045121
]
Hadoop QA commented on HDFS-11907:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
1s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 157 unchanged - 0 fixed = 158 total (was 157) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 31s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11907 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872340/HDFS-11907.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 18ca1e176c4a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 325163f |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/19855/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19855/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19855/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19855/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add metric for time taken by NameNode resource check
>
>
> Key: HDFS-11907
> URL: https://issues.apache.org/jira/browse/HDFS-11907
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments:
[jira] [Commented] (HDFS-11907) Add metric for time taken by NameNode resource check
[ https://issues.apache.org/jira/browse/HDFS-11907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045116#comment-16045116 ] Andrew Wang commented on HDFS-11907: LGTM +1, thanks Chen, Arpit! > Add metric for time taken by NameNode resource check > > > Key: HDFS-11907 > URL: https://issues.apache.org/jira/browse/HDFS-11907 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11907.001.patch, HDFS-11907.002.patch, > HDFS-11907.003.patch, HDFS-11907.004.patch, HDFS-11907.005.patch, > HDFS-11907.006.patch > > > Add a metric to measure the time taken by the NameNode Resource Check. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11907) Add metric for time taken by NameNode resource check
[
https://issues.apache.org/jira/browse/HDFS-11907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-11907:
-
Summary: Add metric for time taken by NameNode resource check (was:
NameNodeResourceChecker should avoid calling df.getAvailable too frequently)
> Add metric for time taken by NameNode resource check
>
>
> Key: HDFS-11907
> URL: https://issues.apache.org/jira/browse/HDFS-11907
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11907.001.patch, HDFS-11907.002.patch,
> HDFS-11907.003.patch, HDFS-11907.004.patch, HDFS-11907.005.patch,
> HDFS-11907.006.patch
>
>
> Currently, {{HealthMonitor#doHealthChecks}} invokes
> {{NameNode#monitorHealth}} which ends up invoking
> {{NameNodeResourceChecker#isResourceAvailable}}, at the frequency of once per
> second by default. And NameNodeResourceChecker#isResourceAvailable invokes
> {{df.getAvailable();}} every time it is called.
> Since available space information should rarely be changing dramatically at
> the pace of per second. A cached value should be sufficient. i.e. only try to
> get the updated value when the cached value is too old. otherwise simply
> return the cached value. This way df.getAvailable() gets invoked less.
> Thanks [~arpitagarwal] for the offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11907) Add metric for time taken by NameNode resource check
[
https://issues.apache.org/jira/browse/HDFS-11907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-11907:
-
Description: Add a metric to measure the time taken by the NameNode
Resource Check. (was: Currently, {{HealthMonitor#doHealthChecks}} invokes
{{NameNode#monitorHealth}} which ends up invoking
{{NameNodeResourceChecker#isResourceAvailable}}, at the frequency of once per
second by default. And NameNodeResourceChecker#isResourceAvailable invokes
{{df.getAvailable();}} every time it is called.
Since available space information should rarely be changing dramatically at the
pace of per second. A cached value should be sufficient. i.e. only try to get
the updated value when the cached value is too old. otherwise simply return the
cached value. This way df.getAvailable() gets invoked less.
Thanks [~arpitagarwal] for the offline discussion.)
> Add metric for time taken by NameNode resource check
>
>
> Key: HDFS-11907
> URL: https://issues.apache.org/jira/browse/HDFS-11907
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11907.001.patch, HDFS-11907.002.patch,
> HDFS-11907.003.patch, HDFS-11907.004.patch, HDFS-11907.005.patch,
> HDFS-11907.006.patch
>
>
> Add a metric to measure the time taken by the NameNode Resource Check.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11907) NameNodeResourceChecker should avoid calling df.getAvailable too frequently
[
https://issues.apache.org/jira/browse/HDFS-11907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045105#comment-16045105
]
Arpit Agarwal commented on HDFS-11907:
--
+1 for the v6 patch. Thanks Chen.
[~andrew.wang] do you have any comments on the latest patch?
> NameNodeResourceChecker should avoid calling df.getAvailable too frequently
> ---
>
> Key: HDFS-11907
> URL: https://issues.apache.org/jira/browse/HDFS-11907
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11907.001.patch, HDFS-11907.002.patch,
> HDFS-11907.003.patch, HDFS-11907.004.patch, HDFS-11907.005.patch,
> HDFS-11907.006.patch
>
>
> Currently, {{HealthMonitor#doHealthChecks}} invokes
> {{NameNode#monitorHealth}} which ends up invoking
> {{NameNodeResourceChecker#isResourceAvailable}}, at the frequency of once per
> second by default. And NameNodeResourceChecker#isResourceAvailable invokes
> {{df.getAvailable();}} every time it is called.
> Since available space information should rarely be changing dramatically at
> the pace of per second. A cached value should be sufficient. i.e. only try to
> get the updated value when the cached value is too old. otherwise simply
> return the cached value. This way df.getAvailable() gets invoked less.
> Thanks [~arpitagarwal] for the offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11907) NameNodeResourceChecker should avoid calling df.getAvailable too frequently
[
https://issues.apache.org/jira/browse/HDFS-11907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-11907:
--
Attachment: HDFS-11907.006.patch
Thanks [~arpitagarwal] for pointing out offline that,
{{checkAvailableResources}} has more than one callers. Post v006 patch to move
the measurement into this method to capture all the callers.
> NameNodeResourceChecker should avoid calling df.getAvailable too frequently
> ---
>
> Key: HDFS-11907
> URL: https://issues.apache.org/jira/browse/HDFS-11907
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11907.001.patch, HDFS-11907.002.patch,
> HDFS-11907.003.patch, HDFS-11907.004.patch, HDFS-11907.005.patch,
> HDFS-11907.006.patch
>
>
> Currently, {{HealthMonitor#doHealthChecks}} invokes
> {{NameNode#monitorHealth}} which ends up invoking
> {{NameNodeResourceChecker#isResourceAvailable}}, at the frequency of once per
> second by default. And NameNodeResourceChecker#isResourceAvailable invokes
> {{df.getAvailable();}} every time it is called.
> Since available space information should rarely be changing dramatically at
> the pace of per second. A cached value should be sufficient. i.e. only try to
> get the updated value when the cached value is too old. otherwise simply
> return the cached value. This way df.getAvailable() gets invoked less.
> Thanks [~arpitagarwal] for the offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11907) NameNodeResourceChecker should avoid calling df.getAvailable too frequently
[
https://issues.apache.org/jira/browse/HDFS-11907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045096#comment-16045096
]
Chen Liang edited comment on HDFS-11907 at 6/9/17 9:59 PM:
---
Thanks [~arpitagarwal] for pointing out offline that,
{{checkAvailableResources}} has more than one callers. Post v006 patch to move
the measurement into this method to capture all the code paths.
was (Author: vagarychen):
Thanks [~arpitagarwal] for pointing out offline that,
{{checkAvailableResources}} has more than one callers. Post v006 patch to move
the measurement into this method to capture all the callers.
> NameNodeResourceChecker should avoid calling df.getAvailable too frequently
> ---
>
> Key: HDFS-11907
> URL: https://issues.apache.org/jira/browse/HDFS-11907
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11907.001.patch, HDFS-11907.002.patch,
> HDFS-11907.003.patch, HDFS-11907.004.patch, HDFS-11907.005.patch,
> HDFS-11907.006.patch
>
>
> Currently, {{HealthMonitor#doHealthChecks}} invokes
> {{NameNode#monitorHealth}} which ends up invoking
> {{NameNodeResourceChecker#isResourceAvailable}}, at the frequency of once per
> second by default. And NameNodeResourceChecker#isResourceAvailable invokes
> {{df.getAvailable();}} every time it is called.
> Since available space information should rarely be changing dramatically at
> the pace of per second. A cached value should be sufficient. i.e. only try to
> get the updated value when the cached value is too old. otherwise simply
> return the cached value. This way df.getAvailable() gets invoked less.
> Thanks [~arpitagarwal] for the offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11960) Successfully closed files can stay under-replicated.
[ https://issues.apache.org/jira/browse/HDFS-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045047#comment-16045047 ] Daryn Sharp commented on HDFS-11960: Looks like a good fix but there really should be a test to prevent a regression. Chasing down all the not-replicating issues has been a pain over the years... > Successfully closed files can stay under-replicated. > > > Key: HDFS-11960 > URL: https://issues.apache.org/jira/browse/HDFS-11960 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Attachments: HDFS-11960.patch > > > If a certain set of conditions hold at the time of a file creation, a block > of the file can stay under-replicated. This is because the block is > mistakenly taken out of the under-replicated block queue and never gets > reevaluated. > Re-evaluation can be triggered if > - a replica containing node dies. > - setrep is called > - NN repl queues are reinitialized (NN failover or restart) > If none of these happens, the block stays under-replicated. > Here is how it happens. > 1) A replica is finalized, but the ACK does not reach the upstream in time. > IBR is also delayed. > 2) A close recovery happens, which updates the gen stamp of "healthy" > replicas. > 3) The file is closed with the healthy replicas. It is added to the > replication queue. > 4) A replication is scheduled, so it is added to the pending replication > list. The replication target is picked as the failed node in 1). > 5) The old IBR is finally received for the failed/excluded node. In the > meantime, the replication fails, because there is already a finalized replica > (with older gen stamp) on the node. > 6) The IBR processing removes the block from the pending list, adds it to > corrupt replicas list, and then issues invalidation. Since the block is in > neither replication queue nor pending list, it stays under-replicated. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11939) Ozone : add read/write random access to Chunks of a key
[ https://issues.apache.org/jira/browse/HDFS-11939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11939: -- Attachment: HDFS-11939-HDFS-7240.002.patch Post v002 patch to fix checkstyle warnining. > Ozone : add read/write random access to Chunks of a key > --- > > Key: HDFS-11939 > URL: https://issues.apache.org/jira/browse/HDFS-11939 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11939-HDFS-7240.001.patch, > HDFS-11939-HDFS-7240.002.patch > > > In Ozone, the value of a key is a sequence of container chunks. Currently, > the only way to read/write the chunks is by using ChunkInputStream and > ChunkOutputStream. However, by the nature of streams, these classes are > currently implemented to only allow sequential read/write. > Ideally we would like to support random access of the chunks. For example, we > want to be able to seek to a specific offset and read/write some data. This > will be critical for key range read/write feature, and potentially important > for supporting parallel read/write. > This JIRA tracks adding support by implementing FileChannel class on top > Chunks. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11804) KMS client needs retry logic
[
https://issues.apache.org/jira/browse/HDFS-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045015#comment-16045015
]
Hadoop QA commented on HDFS-11804:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
18s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 19
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 47s{color} | {color:orange} root: The patch generated 6 new + 27 unchanged -
0 fixed = 33 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 13s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.conf.TestCommonConfigurationFields |
| | hadoop.crypto.key.kms.TestLoadBalancingKMSClientProvider |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestAclsEndToEnd |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11804 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872309/HDFS-11804-trunk-5.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 8956b53b7214 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 99634d1 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/19853/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
|
| checkstyle |
[jira] [Commented] (HDFS-11960) Successfully closed files can stay under-replicated.
[
https://issues.apache.org/jira/browse/HDFS-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045012#comment-16045012
]
Kihwal Lee commented on HDFS-11960:
---
All failed tests were fine when reran.
{noformat}
---
T E S T S
---
Running org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.489 sec
- in org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 81.012 sec
- in org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
Running
org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
Tests run: 14, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 78.64 sec
- in
org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
{noformat}
> Successfully closed files can stay under-replicated.
>
>
> Key: HDFS-11960
> URL: https://issues.apache.org/jira/browse/HDFS-11960
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Critical
> Attachments: HDFS-11960.patch
>
>
> If a certain set of conditions hold at the time of a file creation, a block
> of the file can stay under-replicated. This is because the block is
> mistakenly taken out of the under-replicated block queue and never gets
> reevaluated.
> Re-evaluation can be triggered if
> - a replica containing node dies.
> - setrep is called
> - NN repl queues are reinitialized (NN failover or restart)
> If none of these happens, the block stays under-replicated.
> Here is how it happens.
> 1) A replica is finalized, but the ACK does not reach the upstream in time.
> IBR is also delayed.
> 2) A close recovery happens, which updates the gen stamp of "healthy"
> replicas.
> 3) The file is closed with the healthy replicas. It is added to the
> replication queue.
> 4) A replication is scheduled, so it is added to the pending replication
> list. The replication target is picked as the failed node in 1).
> 5) The old IBR is finally received for the failed/excluded node. In the
> meantime, the replication fails, because there is already a finalized replica
> (with older gen stamp) on the node.
> 6) The IBR processing removes the block from the pending list, adds it to
> corrupt replicas list, and then issues invalidation. Since the block is in
> neither replication queue nor pending list, it stays under-replicated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11939) Ozone : add read/write random access to Chunks of a key
[
https://issues.apache.org/jira/browse/HDFS-11939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044989#comment-16044989
]
Hadoop QA commented on HDFS-11939:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
53s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
31s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
49s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 1 new
+ 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
18s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}129m 31s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
36s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}169m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
| | Inconsistent synchronization of
org.apache.hadoop.scm.storage.ChunkOutputChannel.buffer; locked 64% of time
Unsynchronized access at ChunkOutputChannel.java:64% of time Unsynchronized
access at ChunkOutputChannel.java:[line 230] |
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestErasureCodeBenchmarkThroughput |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality |
| | hadoop.ozone.scm.node.TestNodeManager |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | hadoop.ozone.container.ozoneimpl.TestRatisManager |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
| | org.apache.hadoop.cblock.TestLocalBlockCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11939 |
| JIRA Patch URL |
[jira] [Comment Edited] (HDFS-11907) NameNodeResourceChecker should avoid calling df.getAvailable too frequently
[
https://issues.apache.org/jira/browse/HDFS-11907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044987#comment-16044987
]
Chen Liang edited comment on HDFS-11907 at 6/9/17 8:37 PM:
---
Thanks [~andrew.wang] for the comments. Based on Andrew's comments, instead of
making the change to cache the value, post v005 patch to add a metric to keep
track of available resource check time.
was (Author: vagarychen):
Thanks [~andrew.wang] for the comments. Based on Andrew's comments, instead of
making the cache the value, post v005 patch to add a metric to keep track of
available resource check time.
> NameNodeResourceChecker should avoid calling df.getAvailable too frequently
> ---
>
> Key: HDFS-11907
> URL: https://issues.apache.org/jira/browse/HDFS-11907
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11907.001.patch, HDFS-11907.002.patch,
> HDFS-11907.003.patch, HDFS-11907.004.patch, HDFS-11907.005.patch
>
>
> Currently, {{HealthMonitor#doHealthChecks}} invokes
> {{NameNode#monitorHealth}} which ends up invoking
> {{NameNodeResourceChecker#isResourceAvailable}}, at the frequency of once per
> second by default. And NameNodeResourceChecker#isResourceAvailable invokes
> {{df.getAvailable();}} every time it is called.
> Since available space information should rarely be changing dramatically at
> the pace of per second. A cached value should be sufficient. i.e. only try to
> get the updated value when the cached value is too old. otherwise simply
> return the cached value. This way df.getAvailable() gets invoked less.
> Thanks [~arpitagarwal] for the offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11907) NameNodeResourceChecker should avoid calling df.getAvailable too frequently
[
https://issues.apache.org/jira/browse/HDFS-11907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-11907:
--
Attachment: HDFS-11907.005.patch
Thanks [~andrew.wang] for the comments. Based on Andrew's comments, instead of
making the cache the value, post v005 patch to add a metric to keep track of
available resource check time.
> NameNodeResourceChecker should avoid calling df.getAvailable too frequently
> ---
>
> Key: HDFS-11907
> URL: https://issues.apache.org/jira/browse/HDFS-11907
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11907.001.patch, HDFS-11907.002.patch,
> HDFS-11907.003.patch, HDFS-11907.004.patch, HDFS-11907.005.patch
>
>
> Currently, {{HealthMonitor#doHealthChecks}} invokes
> {{NameNode#monitorHealth}} which ends up invoking
> {{NameNodeResourceChecker#isResourceAvailable}}, at the frequency of once per
> second by default. And NameNodeResourceChecker#isResourceAvailable invokes
> {{df.getAvailable();}} every time it is called.
> Since available space information should rarely be changing dramatically at
> the pace of per second. A cached value should be sufficient. i.e. only try to
> get the updated value when the cached value is too old. otherwise simply
> return the cached value. This way df.getAvailable() gets invoked less.
> Thanks [~arpitagarwal] for the offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11960) Successfully closed files can stay under-replicated.
[
https://issues.apache.org/jira/browse/HDFS-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044985#comment-16044985
]
Hadoop QA commented on HDFS-11960:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 6s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11960 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872320/HDFS-11960.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux dbe1b3b56c89 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 99634d1 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19854/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19854/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19854/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Successfully closed files can stay under-replicated.
>
>
> Key: HDFS-11960
> URL: https://issues.apache.org/jira/browse/HDFS-11960
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority:
[jira] [Commented] (HDFS-11670) [SPS]: Add storagepolicy command to check the status of storage policy Satisfier
[
https://issues.apache.org/jira/browse/HDFS-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044923#comment-16044923
]
Rakesh R commented on HDFS-11670:
-
We can make it simple command {{-isSPSRunning}} -> result could be {{yes}} or
{{no}}. Thanks Uma for the offline discussion.
> [SPS]: Add storagepolicy command to check the status of storage policy
> Satisfier
>
>
> Key: HDFS-11670
> URL: https://issues.apache.org/jira/browse/HDFS-11670
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-11670-HDFS-10285.001.patch
>
>
> Its good to have one command to check SPS is enabled or not. Based on this
> user can take the decision to run the Mover.
> {noformat}
> hdfs storagepolicies -policysatisfierstatus
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11958) Ozone: Ensure KSM is initiated using ProtobufRpcEngine
[
https://issues.apache.org/jira/browse/HDFS-11958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044877#comment-16044877
]
Hadoop QA commented on HDFS-11958:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}112m 55s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.ozone.scm.TestXceiverClientManager |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.ozone.scm.node.TestNodeManager |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.ozone.container.ozoneimpl.TestRatisManager |
| Timed out junit tests | org.apache.hadoop.cblock.TestLocalBlockCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11958 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872284/HDFS-11958-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7d5c1367211e 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11
16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 2ec6464 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19851/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19851/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19851/console |
| Powered by | Apache
[jira] [Updated] (HDFS-11960) Successfully closed files can stay under-replicated.
[ https://issues.apache.org/jira/browse/HDFS-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11960: -- Attachment: HDFS-11960.patch > Successfully closed files can stay under-replicated. > > > Key: HDFS-11960 > URL: https://issues.apache.org/jira/browse/HDFS-11960 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Attachments: HDFS-11960.patch > > > If a certain set of conditions hold at the time of a file creation, a block > of the file can stay under-replicated. This is because the block is > mistakenly taken out of the under-replicated block queue and never gets > reevaluated. > Re-evaluation can be triggered if > - a replica containing node dies. > - setrep is called > - NN repl queues are reinitialized (NN failover or restart) > If none of these happens, the block stays under-replicated. > Here is how it happens. > 1) A replica is finalized, but the ACK does not reach the upstream in time. > IBR is also delayed. > 2) A close recovery happens, which updates the gen stamp of "healthy" > replicas. > 3) The file is closed with the healthy replicas. It is added to the > replication queue. > 4) A replication is scheduled, so it is added to the pending replication > list. The replication target is picked as the failed node in 1). > 5) The old IBR is finally received for the failed/excluded node. In the > meantime, the replication fails, because there is already a finalized replica > (with older gen stamp) on the node. > 6) The IBR processing removes the block from the pending list, adds it to > corrupt replicas list, and then issues invalidation. Since the block is in > neither replication queue nor pending list, it stays under-replicated. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11960) Successfully closed files can stay under-replicated.
[ https://issues.apache.org/jira/browse/HDFS-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11960: -- Status: Patch Available (was: Open) > Successfully closed files can stay under-replicated. > > > Key: HDFS-11960 > URL: https://issues.apache.org/jira/browse/HDFS-11960 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Attachments: HDFS-11960.patch > > > If a certain set of conditions hold at the time of a file creation, a block > of the file can stay under-replicated. This is because the block is > mistakenly taken out of the under-replicated block queue and never gets > reevaluated. > Re-evaluation can be triggered if > - a replica containing node dies. > - setrep is called > - NN repl queues are reinitialized (NN failover or restart) > If none of these happens, the block stays under-replicated. > Here is how it happens. > 1) A replica is finalized, but the ACK does not reach the upstream in time. > IBR is also delayed. > 2) A close recovery happens, which updates the gen stamp of "healthy" > replicas. > 3) The file is closed with the healthy replicas. It is added to the > replication queue. > 4) A replication is scheduled, so it is added to the pending replication > list. The replication target is picked as the failed node in 1). > 5) The old IBR is finally received for the failed/excluded node. In the > meantime, the replication fails, because there is already a finalized replica > (with older gen stamp) on the node. > 6) The IBR processing removes the block from the pending list, adds it to > corrupt replicas list, and then issues invalidation. Since the block is in > neither replication queue nor pending list, it stays under-replicated. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11957) Enable POSIX ACL inheritance by default
[ https://issues.apache.org/jira/browse/HDFS-11957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044847#comment-16044847 ] John Zhuge commented on HDFS-11957: --- Yes [~vagarychen], ACL test failures are related. > Enable POSIX ACL inheritance by default > --- > > Key: HDFS-11957 > URL: https://issues.apache.org/jira/browse/HDFS-11957 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0-alpha2 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HDFS-11957.001.patch > > > It is time to enable POSIX ACL inheritance by default. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7240) Object store in HDFS
[ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044831#comment-16044831 ] Anu Engineer commented on HDFS-7240: I have opened an ozone channel at the ASF slack. Since we are deploying and testing ozone, real time communication is very useful to notify issues as we see them. Please signup at the following page using your apache ID, and join the #Ozone channel if you would like to get notified about how the testing and deployment is going for ozone. https://the-asf.slack.com/signup > Object store in HDFS > > > Key: HDFS-7240 > URL: https://issues.apache.org/jira/browse/HDFS-7240 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: Ozone-architecture-v1.pdf, Ozonedesignupdate.pdf, > ozone_user_v0.pdf > > > This jira proposes to add object store capabilities into HDFS. > As part of the federation work (HDFS-1052) we separated block storage as a > generic storage layer. Using the Block Pool abstraction, new kinds of > namespaces can be built on top of the storage layer i.e. datanodes. > In this jira I will explore building an object store using the datanode > storage, but independent of namespace metadata. > I will soon update with a detailed design document. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11804) KMS client needs retry logic
[ https://issues.apache.org/jira/browse/HDFS-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-11804: -- Attachment: HDFS-11804-trunk-5.patch > KMS client needs retry logic > > > Key: HDFS-11804 > URL: https://issues.apache.org/jira/browse/HDFS-11804 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, > HDFS-11804-trunk-3.patch, HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, > HDFS-11804-trunk.patch > > > The kms client appears to have no retry logic – at all. It's completely > decoupled from the ipc retry logic. This has major impacts if the KMS is > unreachable for any reason, including but not limited to network connection > issues, timeouts, the +restart during an upgrade+. > This has some major ramifications: > # Jobs may fail to submit, although oozie resubmit logic should mask it > # Non-oozie launchers may experience higher rates if they do not already have > retry logic. > # Tasks reading EZ files will fail, probably be masked by framework reattempts > # EZ file creation fails after creating a 0-length file – client receives > EDEK in the create response, then fails when decrypting the EDEK > # Bulk hadoop fs copies, and maybe distcp, will prematurely fail -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11804) KMS client needs retry logic
[ https://issues.apache.org/jira/browse/HDFS-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-11804: -- Status: Patch Available (was: Open) > KMS client needs retry logic > > > Key: HDFS-11804 > URL: https://issues.apache.org/jira/browse/HDFS-11804 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, > HDFS-11804-trunk-3.patch, HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, > HDFS-11804-trunk.patch > > > The kms client appears to have no retry logic – at all. It's completely > decoupled from the ipc retry logic. This has major impacts if the KMS is > unreachable for any reason, including but not limited to network connection > issues, timeouts, the +restart during an upgrade+. > This has some major ramifications: > # Jobs may fail to submit, although oozie resubmit logic should mask it > # Non-oozie launchers may experience higher rates if they do not already have > retry logic. > # Tasks reading EZ files will fail, probably be masked by framework reattempts > # EZ file creation fails after creating a 0-length file – client receives > EDEK in the create response, then fails when decrypting the EDEK > # Bulk hadoop fs copies, and maybe distcp, will prematurely fail -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11804) KMS client needs retry logic
[
https://issues.apache.org/jira/browse/HDFS-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044816#comment-16044816
]
Rushabh S Shah commented on HDFS-11804:
---
Thanks [~xiaochen] for the review.
bq. Not introduced by this, but the first parameter URI providerUri is not used
in KMSCP#createProvider.
Fixed in the latest patch.
bq. Config names should follow existing pattern:
s/kms.client/hadoop.security.kms.client/g
Fixed in the latest patch.
bq. core-default.xml needs to be updated with the new configs
Added in the latest patch.
bq. LBKMSCP, can we bring throw new IOException("No providers configured !");
forward, to add a check at the beginning?
Added in the latest patch.
{quote}
Regarding AuthenticatedException, I think below is 1 possible way to have
LBKMSCP#doOp end up catching one:
KMSCP#createKey -> KMSCP#createConnection ->
DelegationTokenAuthenticatedURL#openConnection
{quote}
For this particular case, {{FailoverOnNetworkExceptionRetry}} retry policy will
do the right thing.
{code:title=FailoverOnNetworkExceptionRetry.java|borderStyle=solid}
// Some comments here
public RetryAction shouldRetry(Exception e, int retries,
int failovers, boolean isIdempotentOrAtMostOnce) throws Exception {
...
...
else if (e instanceof SocketException
|| (e instanceof IOException && !(e instanceof RemoteException))) {
if (isIdempotentOrAtMostOnce) {
return new RetryAction(RetryAction.RetryDecision.FAILOVER_AND_RETRY,
getFailoverOrRetrySleepTime(retries));
} else {
return new RetryAction(RetryAction.RetryDecision.FAIL, 0,
"the invoked method is not idempotent, and unable to determine "
+ "whether it was invoked");
}
...
...
}
{code}
Since we are always passing {{isIdempotentOrAtMostOnce}} flag as false, the
retryPolicy will fail.
There are many places in KMSClientProvider in which all the Exception are
getting converted to IOException.
I don't understand the reasoning behind that.
If we want to fix it, we can open a new jira as that change is outside the
scope of this jira.
Please review the latest patch.
> KMS client needs retry logic
>
>
> Key: HDFS-11804
> URL: https://issues.apache.org/jira/browse/HDFS-11804
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Attachments: HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch,
> HDFS-11804-trunk-3.patch, HDFS-11804-trunk-4.patch, HDFS-11804-trunk.patch
>
>
> The kms client appears to have no retry logic – at all. It's completely
> decoupled from the ipc retry logic. This has major impacts if the KMS is
> unreachable for any reason, including but not limited to network connection
> issues, timeouts, the +restart during an upgrade+.
> This has some major ramifications:
> # Jobs may fail to submit, although oozie resubmit logic should mask it
> # Non-oozie launchers may experience higher rates if they do not already have
> retry logic.
> # Tasks reading EZ files will fail, probably be masked by framework reattempts
> # EZ file creation fails after creating a 0-length file – client receives
> EDEK in the create response, then fails when decrypting the EDEK
> # Bulk hadoop fs copies, and maybe distcp, will prematurely fail
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11933) The function composePolicyName should judge arguments not NULL
[
https://issues.apache.org/jira/browse/HDFS-11933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044803#comment-16044803
]
Chen Liang commented on HDFS-11933:
---
Thanks [~figo] for the patch. v001 patch looks reasonable to me, I wonder
though, have you encountered cases where {{composePolicyName}} was given
invalid arguments? I checked some (but not all) of the callers, seems at least
{{ECSchema schema}} won't be null for these places.
> The function composePolicyName should judge arguments not NULL
> ---
>
> Key: HDFS-11933
> URL: https://issues.apache.org/jira/browse/HDFS-11933
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 3.0.0-alpha3
>Reporter: lufei
>Priority: Minor
> Attachments: HDFS-11933.001.patch
>
>
> Function composePolicyName is called by ErasureCodingPolicy, but both of them
> are not judge the arguments not NULL.It 's better to judge in the function
> composePolicyName.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11958) Ozone: Ensure KSM is initiated using ProtobufRpcEngine
[
https://issues.apache.org/jira/browse/HDFS-11958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-11958:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-7240
Status: Resolved (was: Patch Available)
[~cheersyang] Thanks for the contribution. I have committed this to the feature
branch
> Ozone: Ensure KSM is initiated using ProtobufRpcEngine
> --
>
> Key: HDFS-11958
> URL: https://issues.apache.org/jira/browse/HDFS-11958
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Fix For: HDFS-7240
>
> Attachments: HDFS-11958-HDFS-7240.001.patch
>
>
> Reproduce Steps
> # Launch an ozone cluster
> # Create a volume via commandline
> {code}
> hdfs oz -createVolume http://ozone1.fyre.ibm.com:9864/volume-wwei-0 -user root
> {code}
> it failed with following error
> {noformat}
> SEVERE: The RuntimeException could not be mapped to a response, re-throwing
> to the HTTP container
> java.lang.RuntimeException: java.lang.NoSuchFieldException: versionID
> at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:182)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invocation.(WritableRpcEngine.java:114)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:247)
> at com.sun.proxy.$Proxy18.createVolume(Unknown Source)
> ...
> Caused by: java.lang.NoSuchFieldException: versionID
> at java.lang.Class.getField(Class.java:1703)
> at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:178)
> ... 25 more
> {noformat}
> This was because {{keySpaceManagerClient}} in {{ObjectStoreHandler}}
> currently is not properly initiated, it should be using {{ProtobufRpcEngine}}
> instead of {{WritableRpcEngine}} which is deprecated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11845) Ozone: Output error when DN handshakes with SCM
[
https://issues.apache.org/jira/browse/HDFS-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-11845:
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
[~cheersyang] Thank you for spotting and fixing this issue. I have committed
this to the feature branch.
> Ozone: Output error when DN handshakes with SCM
> ---
>
> Key: HDFS-11845
> URL: https://issues.apache.org/jira/browse/HDFS-11845
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: HDFS-11845-HDFS-7240.001.patch
>
>
> When start SCM and DN, there is always an error in SCM log
> {noformat}
> 17/05/17 15:19:59 WARN ipc.Server: IPC Server handler 9 on 9861, call Call#4
> Retry#0
> org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol.getVersion
> from 172.16.165.133:44824: output error
> 17/05/17 15:19:59 INFO ipc.Server: IPC Server handler 9 on 9861 caught an
> exception
> java.nio.channels.ClosedChannelException
> at
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:461)
> at org.apache.hadoop.ipc.Server.channelWrite(Server.java:3216)
> at org.apache.hadoop.ipc.Server.access$1600(Server.java:135)
> at
> org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:1463)
> at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:1533)
> at
> org.apache.hadoop.ipc.Server$Connection.sendResponse(Server.java:2581)
> at org.apache.hadoop.ipc.Server$Connection.access$300(Server.java:1605)
> at org.apache.hadoop.ipc.Server$RpcCall.doResponse(Server.java:931)
> at org.apache.hadoop.ipc.Server$Call.sendResponse(Server.java:765)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:813)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2659)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11957) Enable POSIX ACL inheritance by default
[ https://issues.apache.org/jira/browse/HDFS-11957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044779#comment-16044779 ] Chen Liang commented on HDFS-11957: --- Thanks [~jzhuge] for the patch. The change itself in v001 patch LGTM, but seems the test failures are related. I only quickly checked TestAclCLI locally, seems the patch might indeed have conflicted the behaviour this test is currently expecting. > Enable POSIX ACL inheritance by default > --- > > Key: HDFS-11957 > URL: https://issues.apache.org/jira/browse/HDFS-11957 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0-alpha2 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HDFS-11957.001.patch > > > It is time to enable POSIX ACL inheritance by default. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11960) Successfully closed files can stay under-replicated.
[
https://issues.apache.org/jira/browse/HDFS-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044775#comment-16044775
]
Kihwal Lee commented on HDFS-11960:
---
The simplest fix will be not letting {{addBlock()}} remove a pending
replication, if the reported genstamp is not current.
{code}
-if (storedBlock != null) {
+if (storedBlock != null &&
+ block.getGenerationStamp() == storedBlock.getGenerationStamp()) {
pendingReconstruction.decrement(storedBlock, node);
}
{code}
This way, the corrupt replica will still be deleted and if the replication is
tried and fails before the deletion, the pending replication will expire and
rescheduled. Even if it is scheduled to the same target again, it will work.
> Successfully closed files can stay under-replicated.
>
>
> Key: HDFS-11960
> URL: https://issues.apache.org/jira/browse/HDFS-11960
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Critical
>
> If a certain set of conditions hold at the time of a file creation, a block
> of the file can stay under-replicated. This is because the block is
> mistakenly taken out of the under-replicated block queue and never gets
> reevaluated.
> Re-evaluation can be triggered if
> - a replica containing node dies.
> - setrep is called
> - NN repl queues are reinitialized (NN failover or restart)
> If none of these happens, the block stays under-replicated.
> Here is how it happens.
> 1) A replica is finalized, but the ACK does not reach the upstream in time.
> IBR is also delayed.
> 2) A close recovery happens, which updates the gen stamp of "healthy"
> replicas.
> 3) The file is closed with the healthy replicas. It is added to the
> replication queue.
> 4) A replication is scheduled, so it is added to the pending replication
> list. The replication target is picked as the failed node in 1).
> 5) The old IBR is finally received for the failed/excluded node. In the
> meantime, the replication fails, because there is already a finalized replica
> (with older gen stamp) on the node.
> 6) The IBR processing removes the block from the pending list, adds it to
> corrupt replicas list, and then issues invalidation. Since the block is in
> neither replication queue nor pending list, it stays under-replicated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11939) Ozone : add read/write random access to Chunks of a key
[ https://issues.apache.org/jira/browse/HDFS-11939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11939: -- Attachment: HDFS-11939-HDFS-7240.001.patch > Ozone : add read/write random access to Chunks of a key > --- > > Key: HDFS-11939 > URL: https://issues.apache.org/jira/browse/HDFS-11939 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11939-HDFS-7240.001.patch > > > In Ozone, the value of a key is a sequence of container chunks. Currently, > the only way to read/write the chunks is by using ChunkInputStream and > ChunkOutputStream. However, by the nature of streams, these classes are > currently implemented to only allow sequential read/write. > Ideally we would like to support random access of the chunks. For example, we > want to be able to seek to a specific offset and read/write some data. This > will be critical for key range read/write feature, and potentially important > for supporting parallel read/write. > This JIRA tracks adding support by implementing FileChannel class on top > Chunks. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11939) Ozone : add read/write random access to Chunks of a key
[ https://issues.apache.org/jira/browse/HDFS-11939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11939: -- Status: Patch Available (was: Open) > Ozone : add read/write random access to Chunks of a key > --- > > Key: HDFS-11939 > URL: https://issues.apache.org/jira/browse/HDFS-11939 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11939-HDFS-7240.001.patch > > > In Ozone, the value of a key is a sequence of container chunks. Currently, > the only way to read/write the chunks is by using ChunkInputStream and > ChunkOutputStream. However, by the nature of streams, these classes are > currently implemented to only allow sequential read/write. > Ideally we would like to support random access of the chunks. For example, we > want to be able to seek to a specific offset and read/write some data. This > will be critical for key range read/write feature, and potentially important > for supporting parallel read/write. > This JIRA tracks adding support by implementing FileChannel class on top > Chunks. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11960) Successfully closed files can stay under-replicated.
[
https://issues.apache.org/jira/browse/HDFS-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044755#comment-16044755
]
Kihwal Lee commented on HDFS-11960:
---
Details of the step 6).
{{processIncrementalBlockReport()}} calls {{addBlock()}} for the received IBR
with the old gen stamp. {{addBlock()}} unconditionally decrements pending count
for the block.
{code:java}
void addBlock(DatanodeStorageInfo storageInfo, Block block, String delHint)
throws IOException {
...
//
// Modify the blocks->datanode map and node's map.
//
pendingReplications.decrement(block, node);
processAndHandleReportedBlock(storageInfo, block, ReplicaState.FINALIZED,
delHintNode);
}
{code}
In {{processAndHandleReportedBlock()}}, the replica is identified as corrupt,
so {{markBlockAsCorrupt()}} is called.
{code}
private void markBlockAsCorrupt(BlockToMarkCorrupt b,
DatanodeStorageInfo storageInfo,
DatanodeDescriptor node) throws IOException {
...
boolean corruptedDuringWrite = minReplicationSatisfied &&
(b.stored.getGenerationStamp() > b.corrupted.getGenerationStamp());
// case 1: have enough number of live replicas
// case 2: corrupted replicas + live replicas > Replication factor
// case 3: Block is marked corrupt due to failure while writing. In this
// case genstamp will be different than that of valid block.
// In all these cases we can delete the replica.
// In case of 3, rbw block will be deleted and valid block can be replicated
if (hasEnoughLiveReplicas || hasMoreCorruptReplicas
|| corruptedDuringWrite) {
// the block is over-replicated so invalidate the replicas immediately
invalidateBlock(b, node);
} else if (namesystem.isPopulatingReplQueues()) {
// add the block to neededReplication
updateNeededReplications(b.stored, -1, 0);
}
}
{code}
As shown above, it is considered as "case 3", which causes immediate
invalidation of the corrupt block. No further check on replication is done.
> Successfully closed files can stay under-replicated.
>
>
> Key: HDFS-11960
> URL: https://issues.apache.org/jira/browse/HDFS-11960
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Critical
>
> If a certain set of conditions hold at the time of a file creation, a block
> of the file can stay under-replicated. This is because the block is
> mistakenly taken out of the under-replicated block queue and never gets
> reevaluated.
> Re-evaluation can be triggered if
> - a replica containing node dies.
> - setrep is called
> - NN repl queues are reinitialized (NN failover or restart)
> If none of these happens, the block stays under-replicated.
> Here is how it happens.
> 1) A replica is finalized, but the ACK does not reach the upstream in time.
> IBR is also delayed.
> 2) A close recovery happens, which updates the gen stamp of "healthy"
> replicas.
> 3) The file is closed with the healthy replicas. It is added to the
> replication queue.
> 4) A replication is scheduled, so it is added to the pending replication
> list. The replication target is picked as the failed node in 1).
> 5) The old IBR is finally received for the failed/excluded node. In the
> meantime, the replication fails, because there is already a finalized replica
> (with older gen stamp) on the node.
> 6) The IBR processing removes the block from the pending list, adds it to
> corrupt replicas list, and then issues invalidation. Since the block is in
> neither replication queue nor pending list, it stays under-replicated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11960) Successfully closed files can stay under-replicated.
[ https://issues.apache.org/jira/browse/HDFS-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11960: -- Summary: Successfully closed files can stay under-replicated. (was: Successfully closed file can stay under-replicated.) > Successfully closed files can stay under-replicated. > > > Key: HDFS-11960 > URL: https://issues.apache.org/jira/browse/HDFS-11960 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > > If a certain set of conditions hold at the time of a file creation, a block > of the file can stay under-replicated. This is because the block is > mistakenly taken out of the under-replicated block queue and never gets > reevaluated. > Re-evaluation can be triggered if > - a replica containing node dies. > - setrep is called > - NN repl queues are reinitialized (NN failover or restart) > If none of these happens, the block stays under-replicated. > Here is how it happens. > 1) A replica is finalized, but the ACK does not reach the upstream in time. > IBR is also delayed. > 2) A close recovery happens, which updates the gen stamp of "healthy" > replicas. > 3) The file is closed with the healthy replicas. It is added to the > replication queue. > 4) A replication is scheduled, so it is added to the pending replication > list. The replication target is picked as the failed node in 1). > 5) The old IBR is finally received for the failed/excluded node. In the > meantime, the replication fails, because there is already a finalized replica > (with older gen stamp) on the node. > 6) The IBR processing removes the block from the pending list, adds it to > corrupt replicas list, and then issues invalidation. Since the block is in > neither replication queue nor pending list, it stays under-replicated. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11960) Successfully closed file can stay under-replicated.
[ https://issues.apache.org/jira/browse/HDFS-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-11960: - Assignee: Kihwal Lee > Successfully closed file can stay under-replicated. > --- > > Key: HDFS-11960 > URL: https://issues.apache.org/jira/browse/HDFS-11960 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > > If a certain set of conditions hold at the time of a file creation, a block > of the file can stay under-replicated. This is because the block is > mistakenly taken out of the under-replicated block queue and never gets > reevaluated. > Re-evaluation can be triggered if > - a replica containing node dies. > - setrep is called > - NN repl queues are reinitialized (NN failover or restart) > If none of these happens, the block stays under-replicated. > Here is how it happens. > 1) A replica is finalized, but the ACK does not reach the upstream in time. > IBR is also delayed. > 2) A close recovery happens, which updates the gen stamp of "healthy" > replicas. > 3) The file is closed with the healthy replicas. It is added to the > replication queue. > 4) A replication is scheduled, so it is added to the pending replication > list. The replication target is picked as the failed node in 1). > 5) The old IBR is finally received for the failed/excluded node. In the > meantime, the replication fails, because there is already a finalized replica > (with older gen stamp) on the node. > 6) The IBR processing removes the block from the pending list, adds it to > corrupt replicas list, and then issues invalidation. Since the block is in > neither replication queue nor pending list, it stays under-replicated. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Moved] (HDFS-11960) Successfully closed file can stay under-replicated.
[ https://issues.apache.org/jira/browse/HDFS-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee moved HADOOP-14514 to HDFS-11960: Target Version/s: 2.8.2 (was: 2.8.2) Key: HDFS-11960 (was: HADOOP-14514) Project: Hadoop HDFS (was: Hadoop Common) > Successfully closed file can stay under-replicated. > --- > > Key: HDFS-11960 > URL: https://issues.apache.org/jira/browse/HDFS-11960 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Priority: Critical > > If a certain set of conditions hold at the time of a file creation, a block > of the file can stay under-replicated. This is because the block is > mistakenly taken out of the under-replicated block queue and never gets > reevaluated. > Re-evaluation can be triggered if > - a replica containing node dies. > - setrep is called > - NN repl queues are reinitialized (NN failover or restart) > If none of these happens, the block stays under-replicated. > Here is how it happens. > 1) A replica is finalized, but the ACK does not reach the upstream in time. > IBR is also delayed. > 2) A close recovery happens, which updates the gen stamp of "healthy" > replicas. > 3) The file is closed with the healthy replicas. It is added to the > replication queue. > 4) A replication is scheduled, so it is added to the pending replication > list. The replication target is picked as the failed node in 1). > 5) The old IBR is finally received for the failed/excluded node. In the > meantime, the replication fails, because there is already a finalized replica > (with older gen stamp) on the node. > 6) The IBR processing removes the block from the pending list, adds it to > corrupt replicas list, and then issues invalidation. Since the block is in > neither replication queue nor pending list, it stays under-replicated. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11845) Ozone: Output error when DN handshakes with SCM
[
https://issues.apache.org/jira/browse/HDFS-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044746#comment-16044746
]
Anu Engineer commented on HDFS-11845:
-
+1, I think we set small values for ease of unit testing. You are right, in a
real cluster it is too aggressive. I will commit this shortly.
> Ozone: Output error when DN handshakes with SCM
> ---
>
> Key: HDFS-11845
> URL: https://issues.apache.org/jira/browse/HDFS-11845
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: HDFS-11845-HDFS-7240.001.patch
>
>
> When start SCM and DN, there is always an error in SCM log
> {noformat}
> 17/05/17 15:19:59 WARN ipc.Server: IPC Server handler 9 on 9861, call Call#4
> Retry#0
> org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol.getVersion
> from 172.16.165.133:44824: output error
> 17/05/17 15:19:59 INFO ipc.Server: IPC Server handler 9 on 9861 caught an
> exception
> java.nio.channels.ClosedChannelException
> at
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:461)
> at org.apache.hadoop.ipc.Server.channelWrite(Server.java:3216)
> at org.apache.hadoop.ipc.Server.access$1600(Server.java:135)
> at
> org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:1463)
> at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:1533)
> at
> org.apache.hadoop.ipc.Server$Connection.sendResponse(Server.java:2581)
> at org.apache.hadoop.ipc.Server$Connection.access$300(Server.java:1605)
> at org.apache.hadoop.ipc.Server$RpcCall.doResponse(Server.java:931)
> at org.apache.hadoop.ipc.Server$Call.sendResponse(Server.java:765)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:813)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2659)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11947) When constructing a thread name, BPOfferService may print a bogus warning message
[ https://issues.apache.org/jira/browse/HDFS-11947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044744#comment-16044744 ] Hanisha Koneru commented on HDFS-11947: --- Thanks [~cheersyang] for fixing it. +1 (non-binding) for patch v03. > When constructing a thread name, BPOfferService may print a bogus warning > message > -- > > Key: HDFS-11947 > URL: https://issues.apache.org/jira/browse/HDFS-11947 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang >Priority: Minor > Attachments: HDFS-11947.001.patch, HDFS-11947.002.patch, > HDFS-11947.003.patch > > > HDFS-11558 tries to get Block pool ID for constructing thread names. When > the service is not yet registered with NN, it prints the bogus warning "Block > pool ID needed, but service not yet registered with NN" with stack trace. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11958) Ozone: Ensure KSM is initiated using ProtobufRpcEngine
[
https://issues.apache.org/jira/browse/HDFS-11958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044741#comment-16044741
]
Anu Engineer commented on HDFS-11958:
-
+1, I will commit this shortly.
> Ozone: Ensure KSM is initiated using ProtobufRpcEngine
> --
>
> Key: HDFS-11958
> URL: https://issues.apache.org/jira/browse/HDFS-11958
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Attachments: HDFS-11958-HDFS-7240.001.patch
>
>
> Reproduce Steps
> # Launch an ozone cluster
> # Create a volume via commandline
> {code}
> hdfs oz -createVolume http://ozone1.fyre.ibm.com:9864/volume-wwei-0 -user root
> {code}
> it failed with following error
> {noformat}
> SEVERE: The RuntimeException could not be mapped to a response, re-throwing
> to the HTTP container
> java.lang.RuntimeException: java.lang.NoSuchFieldException: versionID
> at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:182)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invocation.(WritableRpcEngine.java:114)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:247)
> at com.sun.proxy.$Proxy18.createVolume(Unknown Source)
> ...
> Caused by: java.lang.NoSuchFieldException: versionID
> at java.lang.Class.getField(Class.java:1703)
> at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:178)
> ... 25 more
> {noformat}
> This was because {{keySpaceManagerClient}} in {{ObjectStoreHandler}}
> currently is not properly initiated, it should be using {{ProtobufRpcEngine}}
> instead of {{WritableRpcEngine}} which is deprecated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11185) Ozone: remove disabled tests
[ https://issues.apache.org/jira/browse/HDFS-11185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11185: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) [~xyao] Thanks for the reviews. I have committed this to the feature branch. > Ozone: remove disabled tests > > > Key: HDFS-11185 > URL: https://issues.apache.org/jira/browse/HDFS-11185 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-11185-HDFS-7240.001.patch > > > This jira tracks disabling of some ozone tests in HDFS-11184. This is allows > us to separate SCM and KSM into clean modules. > Disabled tests for time being are: > * testLocationsForSingleKey > * testLocationsForMultipleKeys > * testMultipleDataNodes -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11957) Enable POSIX ACL inheritance by default
[
https://issues.apache.org/jira/browse/HDFS-11957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044731#comment-16044731
]
Hadoop QA commented on HDFS-11957:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
54s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 46s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestFSImageWithAcl |
| | hadoop.hdfs.web.TestWebHDFSAcl |
| | hadoop.cli.TestAclCLI |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.server.namenode.TestFileContextAcl |
| | hadoop.hdfs.server.namenode.TestNameNodeAcl |
| Timed out junit tests | org.apache.hadoop.hdfs.TestReplication |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11957 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872277/HDFS-11957.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 455c0a184cf8 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 99634d1 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19850/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19850/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19850/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Enable POSIX ACL inheritance by default
> ---
>
>
[jira] [Commented] (HDFS-11955) Ozone: Set proper parameter default values for listBuckets http request
[
https://issues.apache.org/jira/browse/HDFS-11955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044697#comment-16044697
]
Weiwei Yang commented on HDFS-11955:
Tested on a cluster, when list buckets without any parameter, I get
{noformat}
17/06/09 09:59:39 INFO exceptions.OzoneExceptionMapper:
Returning exception. ex:
{
"httpCode":400,
"shortMessage":"invalidVolumeName",
"resource":"root",
"message":"Illegal max number of keys specified, the value must be in range
(0, 1024], actual : 0.",
"requestID":"157e681b-728c-497d-9635-d232b9ebe093",
"hostName":"ozone1.fyre.ibm.com"
}
{noformat}
Will continue with more testing and upload a complete fix.
> Ozone: Set proper parameter default values for listBuckets http request
> ---
>
> Key: HDFS-11955
> URL: https://issues.apache.org/jira/browse/HDFS-11955
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>
> HDFS-11779 implements the listBuckets function in ozone server side, the API
> supports several parameters, startKey, count and prefix. But both of them are
> optional for the client side rest API. This jira is to make sure we set
> proper default values in the http request if they are not explicitly set by
> users.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11185) Ozone: remove disabled tests
[ https://issues.apache.org/jira/browse/HDFS-11185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11185: Summary: Ozone: remove disabled tests (was: Ozone: Fix disabled tests) > Ozone: remove disabled tests > > > Key: HDFS-11185 > URL: https://issues.apache.org/jira/browse/HDFS-11185 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-11185-HDFS-7240.001.patch > > > This jira tracks disabling of some ozone tests in HDFS-11184. This is allows > us to separate SCM and KSM into clean modules. > Disabled tests for time being are: > * testLocationsForSingleKey > * testLocationsForMultipleKeys > * testMultipleDataNodes -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11959) Ozone: Audit Logs
Weiwei Yang created HDFS-11959: -- Summary: Ozone: Audit Logs Key: HDFS-11959 URL: https://issues.apache.org/jira/browse/HDFS-11959 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Reporter: Weiwei Yang Assignee: Weiwei Yang Add audit logs for ozone components. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11958) Ozone: Ensure KSM is initiated using ProtobufRpcEngine
[
https://issues.apache.org/jira/browse/HDFS-11958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-11958:
---
Status: Patch Available (was: Open)
> Ozone: Ensure KSM is initiated using ProtobufRpcEngine
> --
>
> Key: HDFS-11958
> URL: https://issues.apache.org/jira/browse/HDFS-11958
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Attachments: HDFS-11958-HDFS-7240.001.patch
>
>
> Reproduce Steps
> # Launch an ozone cluster
> # Create a volume via commandline
> {code}
> hdfs oz -createVolume http://ozone1.fyre.ibm.com:9864/volume-wwei-0 -user root
> {code}
> it failed with following error
> {noformat}
> SEVERE: The RuntimeException could not be mapped to a response, re-throwing
> to the HTTP container
> java.lang.RuntimeException: java.lang.NoSuchFieldException: versionID
> at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:182)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invocation.(WritableRpcEngine.java:114)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:247)
> at com.sun.proxy.$Proxy18.createVolume(Unknown Source)
> ...
> Caused by: java.lang.NoSuchFieldException: versionID
> at java.lang.Class.getField(Class.java:1703)
> at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:178)
> ... 25 more
> {noformat}
> This was because {{keySpaceManagerClient}} in {{ObjectStoreHandler}}
> currently is not properly initiated, it should be using {{ProtobufRpcEngine}}
> instead of {{WritableRpcEngine}} which is deprecated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11958) Ozone: Ensure KSM is initiated using ProtobufRpcEngine
[
https://issues.apache.org/jira/browse/HDFS-11958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-11958:
---
Attachment: HDFS-11958-HDFS-7240.001.patch
> Ozone: Ensure KSM is initiated using ProtobufRpcEngine
> --
>
> Key: HDFS-11958
> URL: https://issues.apache.org/jira/browse/HDFS-11958
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Attachments: HDFS-11958-HDFS-7240.001.patch
>
>
> Reproduce Steps
> # Launch an ozone cluster
> # Create a volume via commandline
> {code}
> hdfs oz -createVolume http://ozone1.fyre.ibm.com:9864/volume-wwei-0 -user root
> {code}
> it failed with following error
> {noformat}
> SEVERE: The RuntimeException could not be mapped to a response, re-throwing
> to the HTTP container
> java.lang.RuntimeException: java.lang.NoSuchFieldException: versionID
> at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:182)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invocation.(WritableRpcEngine.java:114)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:247)
> at com.sun.proxy.$Proxy18.createVolume(Unknown Source)
> ...
> Caused by: java.lang.NoSuchFieldException: versionID
> at java.lang.Class.getField(Class.java:1703)
> at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:178)
> ... 25 more
> {noformat}
> This was because {{keySpaceManagerClient}} in {{ObjectStoreHandler}}
> currently is not properly initiated, it should be using {{ProtobufRpcEngine}}
> instead of {{WritableRpcEngine}} which is deprecated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11958) Ozone: Ensure KSM is initiated using ProtobufRpcEngine
Weiwei Yang created HDFS-11958:
--
Summary: Ozone: Ensure KSM is initiated using ProtobufRpcEngine
Key: HDFS-11958
URL: https://issues.apache.org/jira/browse/HDFS-11958
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Weiwei Yang
Assignee: Weiwei Yang
Priority: Critical
Reproduce Steps
# Launch an ozone cluster
# Create a volume via commandline
{code}
hdfs oz -createVolume http://ozone1.fyre.ibm.com:9864/volume-wwei-0 -user root
{code}
it failed with following error
{noformat}
SEVERE: The RuntimeException could not be mapped to a response, re-throwing to
the HTTP container
java.lang.RuntimeException: java.lang.NoSuchFieldException: versionID
at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:182)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invocation.(WritableRpcEngine.java:114)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:247)
at com.sun.proxy.$Proxy18.createVolume(Unknown Source)
...
Caused by: java.lang.NoSuchFieldException: versionID
at java.lang.Class.getField(Class.java:1703)
at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:178)
... 25 more
{noformat}
This was because {{keySpaceManagerClient}} in {{ObjectStoreHandler}} currently
is not properly initiated, it should be using {{ProtobufRpcEngine}} instead of
{{WritableRpcEngine}} which is deprecated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11957) Enable POSIX ACL inheritance by default
[ https://issues.apache.org/jira/browse/HDFS-11957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-11957: -- Attachment: HDFS-11957.001.patch Patch 001 * Update DFS_NAMENODE_POSIX_ACL_INHERITANCE_ENABLED_DEFAULT * Update hdfs-default.xml * Update doc > Enable POSIX ACL inheritance by default > --- > > Key: HDFS-11957 > URL: https://issues.apache.org/jira/browse/HDFS-11957 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0-alpha2 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HDFS-11957.001.patch > > > It is time to enable POSIX ACL inheritance by default. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11957) Enable POSIX ACL inheritance by default
[ https://issues.apache.org/jira/browse/HDFS-11957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-11957: -- Status: Patch Available (was: Open) > Enable POSIX ACL inheritance by default > --- > > Key: HDFS-11957 > URL: https://issues.apache.org/jira/browse/HDFS-11957 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0-alpha2 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HDFS-11957.001.patch > > > It is time to enable POSIX ACL inheritance by default. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11185) Ozone: Fix disabled tests
[ https://issues.apache.org/jira/browse/HDFS-11185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044559#comment-16044559 ] Xiaoyu Yao commented on HDFS-11185: --- Thanks [~anu] for working on this. Patch LGTM. +1. > Ozone: Fix disabled tests > - > > Key: HDFS-11185 > URL: https://issues.apache.org/jira/browse/HDFS-11185 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-11185-HDFS-7240.001.patch > > > This jira tracks disabling of some ozone tests in HDFS-11184. This is allows > us to separate SCM and KSM into clean modules. > Disabled tests for time being are: > * testLocationsForSingleKey > * testLocationsForMultipleKeys > * testMultipleDataNodes -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11957) Enable POSIX ACL inheritance by default
John Zhuge created HDFS-11957: - Summary: Enable POSIX ACL inheritance by default Key: HDFS-11957 URL: https://issues.apache.org/jira/browse/HDFS-11957 Project: Hadoop HDFS Issue Type: Improvement Components: security Affects Versions: 3.0.0-alpha2 Reporter: John Zhuge Assignee: John Zhuge It is time to enable POSIX ACL inheritance by default. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6962) ACL inheritance conflicts with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-6962: - Release Note: The original implementation of HDFS ACLs applied the client's umask to the permissions when inheriting a default ACL defined on a parent directory. This behavior is a deviation from the POSIX ACL specification, which states that the umask has no influence when a default ACL propagates from parent to child. HDFS now offers the capability to ignore the umask in this case for improved compliance with POSIX. This change is considered backward-incompatible, so the new behavior is off by default and must be explicitly configured by setting dfs.namenode.posix.acl.inheritance.enabled to true in hdfs-site.xml. Please see the HDFS Permissions Guide for further details. was: The original implementation of HDFS ACLs applied the client's umask to the permissions when inheriting a default ACL defined on a parent directory. This behavior is a deviation from the POSIX ACL specification, which states that the umask has no influence when a default ACL propagates from parent to child. HDFS now offers the capability to ignore the umask in this case for improved compliance with POSIX. This change is considered backward-incompatible, so the new behavior is off by default and must be explicitly configured by setting dfs.namenode.posix.acl.inheritance.enabled to true in hdfs-site.xml. Please see the HDFS Permissions Guide for further details. > ACL inheritance conflicts with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Fix For: 3.0.0-alpha2 > > Attachments: disabled_new_client.log, disabled_old_client.log, > enabled_new_client.log, enabled_old_client.log, HDFS-6962.001.patch, > HDFS-6962.002.patch, HDFS-6962.003.patch, HDFS-6962.004.patch, > HDFS-6962.005.patch, HDFS-6962.006.patch, HDFS-6962.007.patch, > HDFS-6962.008.patch, HDFS-6962.009.patch, HDFS-6962.010.patch, > HDFS-6962.1.patch, run_compat_tests, run_unit_tests, test_plan.md > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was
[jira] [Updated] (HDFS-6962) ACL inheritance conflicts with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-6962: - Release Note: The original implementation of HDFS ACLs applied the client's umask to the permissions when inheriting a default ACL defined on a parent directory. This behavior is a deviation from the POSIX ACL specification, which states that the umask has no influence when a default ACL propagates from parent to child. HDFS now offers the capability to ignore the umask in this case for improved compliance with POSIX. This change is considered backward-incompatible, so the new behavior is off by default and must be explicitly configured by setting dfs.namenode.posix.acl.inheritance.enabled to true in hdfs-site.xml. Please see the HDFS Permissions Guide for further details. was:The original implementation of HDFS ACLs applied the client's umask to the permissions when inheriting a default ACL defined on a parent directory. This behavior is a deviation from the POSIX ACL specification, which states that the umask has no influence when a default ACL propagates from parent to child. HDFS now offers the capability to ignore the umask in this case for improved compliance with POSIX. This change is considered backward-incompatible, so the new behavior is off by default and must be explicitly configured by setting dfs.namenode.posix.acl.inheritance.enabled to true in hdfs-site.xml. Please see the HDFS Permissions Guide for further details. > ACL inheritance conflicts with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Fix For: 3.0.0-alpha2 > > Attachments: disabled_new_client.log, disabled_old_client.log, > enabled_new_client.log, enabled_old_client.log, HDFS-6962.001.patch, > HDFS-6962.002.patch, HDFS-6962.003.patch, HDFS-6962.004.patch, > HDFS-6962.005.patch, HDFS-6962.006.patch, HDFS-6962.007.patch, > HDFS-6962.008.patch, HDFS-6962.009.patch, HDFS-6962.010.patch, > HDFS-6962.1.patch, run_compat_tests, run_unit_tests, test_plan.md > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message
[jira] [Updated] (HDFS-11804) KMS client needs retry logic
[ https://issues.apache.org/jira/browse/HDFS-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-11804: -- Status: Open (was: Patch Available) > KMS client needs retry logic > > > Key: HDFS-11804 > URL: https://issues.apache.org/jira/browse/HDFS-11804 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, > HDFS-11804-trunk-3.patch, HDFS-11804-trunk-4.patch, HDFS-11804-trunk.patch > > > The kms client appears to have no retry logic – at all. It's completely > decoupled from the ipc retry logic. This has major impacts if the KMS is > unreachable for any reason, including but not limited to network connection > issues, timeouts, the +restart during an upgrade+. > This has some major ramifications: > # Jobs may fail to submit, although oozie resubmit logic should mask it > # Non-oozie launchers may experience higher rates if they do not already have > retry logic. > # Tasks reading EZ files will fail, probably be masked by framework reattempts > # EZ file creation fails after creating a 0-length file – client receives > EDEK in the create response, then fails when decrypting the EDEK > # Bulk hadoop fs copies, and maybe distcp, will prematurely fail -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11951) Ozone: Handle potential inconsistent states while listing keys
[ https://issues.apache.org/jira/browse/HDFS-11951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044513#comment-16044513 ] Yiqun Lin commented on HDFS-11951: -- Thanks for your comment, [~cheersyang]. The comment makes sense to me. > Ozone: Handle potential inconsistent states while listing keys > -- > > Key: HDFS-11951 > URL: https://issues.apache.org/jira/browse/HDFS-11951 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > > In the discussions in HDFS-11779, there might be a corner case we running > into the situation that some key are deleted while we are listing them. For > example > 1. A list call is made say from key1 to key100 is returned. > 2. Someone deletes the key100 > 3. A list continue call is made with startKey=key100. > We need to investigate this case and make sure this is properly handled in > ozone. See more discussion from [here | > https://issues.apache.org/jira/browse/HDFS-11779?focusedCommentId=16042356=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16042356]. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11782) Ozone: KSM: Add listKey
[ https://issues.apache.org/jira/browse/HDFS-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044511#comment-16044511 ] Yiqun Lin commented on HDFS-11782: -- Hi [~anu], I am working on this now and will attach the patch in one or two days, :). > Ozone: KSM: Add listKey > --- > > Key: HDFS-11782 > URL: https://issues.apache.org/jira/browse/HDFS-11782 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: HDFS-7240 >Affects Versions: ozone >Reporter: Anu Engineer >Assignee: Yiqun Lin > > Add support for listing keys in a bucket. Just like other 2 list operations, > this API supports paging via, prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11845) Ozone: Output error when DN handshakes with SCM
[
https://issues.apache.org/jira/browse/HDFS-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044496#comment-16044496
]
Weiwei Yang commented on HDFS-11845:
[~anu], [~xyao] please kindly review. Thanks.
> Ozone: Output error when DN handshakes with SCM
> ---
>
> Key: HDFS-11845
> URL: https://issues.apache.org/jira/browse/HDFS-11845
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: HDFS-11845-HDFS-7240.001.patch
>
>
> When start SCM and DN, there is always an error in SCM log
> {noformat}
> 17/05/17 15:19:59 WARN ipc.Server: IPC Server handler 9 on 9861, call Call#4
> Retry#0
> org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol.getVersion
> from 172.16.165.133:44824: output error
> 17/05/17 15:19:59 INFO ipc.Server: IPC Server handler 9 on 9861 caught an
> exception
> java.nio.channels.ClosedChannelException
> at
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:461)
> at org.apache.hadoop.ipc.Server.channelWrite(Server.java:3216)
> at org.apache.hadoop.ipc.Server.access$1600(Server.java:135)
> at
> org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:1463)
> at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:1533)
> at
> org.apache.hadoop.ipc.Server$Connection.sendResponse(Server.java:2581)
> at org.apache.hadoop.ipc.Server$Connection.access$300(Server.java:1605)
> at org.apache.hadoop.ipc.Server$RpcCall.doResponse(Server.java:931)
> at org.apache.hadoop.ipc.Server$Call.sendResponse(Server.java:765)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:813)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2659)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11945) Internal lease recovery may not be retried for a long time
[ https://issues.apache.org/jira/browse/HDFS-11945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044490#comment-16044490 ] Kihwal Lee commented on HDFS-11945: --- Thanks, [~liuml07]! > Internal lease recovery may not be retried for a long time > -- > > Key: HDFS-11945 > URL: https://issues.apache.org/jira/browse/HDFS-11945 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 3.0.0-alpha4, 2.8.2 > > Attachments: HDFS-11945.branch-2.v2.patch, HDFS-11945.trunk.patch, > HDFS-11945.trunk.v2.patch > > > Lease is assigned per client who is identified by its holder ID or client ID, > thus a renewal or an expiration of a lease affects all files being written by > the client. > When a client/writer dies without closing a file, its lease expires in one > hour (hard limit) and the namenode tries to recover the lease. As a part of > the process, the namenode takes the ownership of the lease and renews it. If > the recovery does not finish successfully, the lease will expire in one hour > and the namenode will try again to recover the lease. > However, if a file system has another lease expiring within the hour, the > recovery attempt for the lease will push forward the expiration of the lease > held by the namenode. This causes failed lease recoveries to be not retried > for a long time. We have seen it happening for days. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[
https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044343#comment-16044343
]
Hadoop QA commented on HDFS-11647:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
6s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
37s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 19
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
15s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 11m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 4s{color} | {color:orange} root: The patch generated 7 new + 146 unchanged
- 0 fixed = 153 total (was 146) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
16s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 12s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
18s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 43s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}147m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Dead store to bsp in
org.apache.hadoop.hdfs.server.namenode.INodeFile.computeContentSummary(int,
ContentSummaryComputationContext) At
INodeFile.java:org.apache.hadoop.hdfs.server.namenode.INodeFile.computeContentSummary(int,
ContentSummaryComputationContext) At INodeFile.java:[line 862] |
| Failed junit tests | hadoop.cli.TestCLI |
| | hadoop.fs.shell.TestCount |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.server.namenode.ha.TestHAFsck |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11647 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-11646) Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable
[
https://issues.apache.org/jira/browse/HDFS-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044336#comment-16044336
]
Hadoop QA commented on HDFS-11646:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
37s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
19s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 19
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 10m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 53s{color} | {color:orange} root: The patch generated 9 new + 160 unchanged
- 0 fixed = 169 total (was 160) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
48s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 11s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
18s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Dead store to bsp in
org.apache.hadoop.hdfs.server.namenode.INodeFile.computeContentSummary(int,
ContentSummaryComputationContext) At
INodeFile.java:org.apache.hadoop.hdfs.server.namenode.INodeFile.computeContentSummary(int,
ContentSummaryComputationContext) At INodeFile.java:[line 862] |
| Failed junit tests | hadoop.fs.TestFsShellReturnCode |
| | hadoop.security.TestRaceWhenRelogin |
| | hadoop.cli.TestCLI |
| | hadoop.fs.shell.TestAclCommands |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.cli.TestAclCLI |
| | hadoop.cli.TestCryptoAdminCLI |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.cli.TestAclCLIWithPosixAclInheritance |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.cli.TestHDFSCLI |
| |
[jira] [Commented] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[
https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044281#comment-16044281
]
Hadoop QA commented on HDFS-11647:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
45s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
23s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 19
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 10m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
8s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 52s{color} | {color:orange} root: The patch generated 7 new + 146 unchanged
- 0 fixed = 153 total (was 146) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
54s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 3s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
28s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 42s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Dead store to bsp in
org.apache.hadoop.hdfs.server.namenode.INodeFile.computeContentSummary(int,
ContentSummaryComputationContext) At
INodeFile.java:org.apache.hadoop.hdfs.server.namenode.INodeFile.computeContentSummary(int,
ContentSummaryComputationContext) At INodeFile.java:[line 862] |
| Failed junit tests | hadoop.cli.TestCLI |
| | hadoop.fs.shell.TestCount |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11647 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872217/HDFS-11647-001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc xml |
| uname | Linux b4eafd73628c
[jira] [Commented] (HDFS-11726) [SPS] : StoragePolicySatisfier should not select same storage type as source and destination in same datanode.
[
https://issues.apache.org/jira/browse/HDFS-11726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044279#comment-16044279
]
Surendra Singh Lilhore commented on HDFS-11726:
---
Thanks [~rakeshr] for review and commit..
Thanks [~umamaheswararao] for review..
> [SPS] : StoragePolicySatisfier should not select same storage type as source
> and destination in same datanode.
> --
>
> Key: HDFS-11726
> URL: https://issues.apache.org/jira/browse/HDFS-11726
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: HDFS-10285
>
> Attachments: HDFS-11726-HDFS-10285.001.patch,
> HDFS-11726-HDFS-10285.002.patch
>
>
> {code}
> 2017-04-30 16:12:28,569 [BlockMoverTask-0] INFO
> datanode.StoragePolicySatisfyWorker (Worker.java:moveBlock(248)) - Start
> moving block:blk_1073741826_1002 from src:127.0.0.1:41699 to
> destin:127.0.0.1:41699 to satisfy storageType, sourceStoragetype:ARCHIVE and
> destinStoragetype:ARCHIVE
> {code}
> {code}
> 2017-04-30 16:12:28,571 [DataXceiver for client /127.0.0.1:36428 [Replacing
> block BP-1409501412-127.0.1.1-1493548923222:blk_1073741826_1002 from
> 6c7aa66e-a778-43d5-89f6-053d5f6b35bc]] INFO datanode.DataNode
> (DataXceiver.java:replaceBlock(1202)) - opReplaceBlock
> BP-1409501412-127.0.1.1-1493548923222:blk_1073741826_1002 received exception
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
> FinalizedReplica, blk_1073741826_1002, FINALIZED
> getNumBytes() = 1024
> getBytesOnDisk() = 1024
> getVisibleLength()= 1024
> getVolume() =
> /home/sachin/software/hadoop/HDFS-10285/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data7
> getBlockURI() =
> file:/home/sachin/software/hadoop/HDFS-10285/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data7/current/BP-1409501412-127.0.1.1-1493548923222/current/finalized/subdir0/subdir0/blk_1073741826
> already exists on storage ARCHIVE
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11711) DN should not delete the block On "Too many open files" Exception
[
https://issues.apache.org/jira/browse/HDFS-11711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044239#comment-16044239
]
Brahma Reddy Battula commented on HDFS-11711:
-
bq.it should just throw a new type of exception in these two cases.
Looks this better, we can have different type of exception .Instead of deleting
on FNFE, Validate the file existence before opening stream, and then throw
different exception..?
> DN should not delete the block On "Too many open files" Exception
> -
>
> Key: HDFS-11711
> URL: https://issues.apache.org/jira/browse/HDFS-11711
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2
>
> Attachments: HDFS-11711-002.patch, HDFS-11711-003.patch,
> HDFS-11711-004.patch, HDFS-11711-branch-2-002.patch,
> HDFS-11711-branch-2-003.patch, HDFS-11711.patch
>
>
> *Seen the following scenario in one of our customer environment*
> * while jobclient writing {{"job.xml"}} there are pipeline failures and
> written to only one DN.
> * when mapper reading the {{"job.xml"}}, DN got {{"Too many open files"}} (as
> system exceed limit) and block got deleted. Hence mapper failed to read and
> job got failed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
How to monitor dfs.datanode.handler.count
Hi Team, We had recently increased datanode handler count in our cluster. But how do we monitor handler count in Cloudera Manager using charts. or any other method. Please advise. Regards, SP
[jira] [Updated] (HDFS-11646) Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable
[ https://issues.apache.org/jira/browse/HDFS-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11646: - Attachment: (was: HADOOP-11646.patch) > Add -E option in 'ls' to list erasure coding policy of each file and > directory if applicable > > > Key: HDFS-11646 > URL: https://issues.apache.org/jira/browse/HDFS-11646 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11646-001.patch, HDFS-11646-002.patch > > > Add -E option in "ls" to show erasure coding policy of file and directory, > leverage the "number_of_replicas " column. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11647: - Attachment: HDFS-11647-002.patch > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11647-001.patch, HDFS-11647-002.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11646) Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable
[ https://issues.apache.org/jira/browse/HDFS-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11646: - Attachment: HDFS-11646-002.patch > Add -E option in 'ls' to list erasure coding policy of each file and > directory if applicable > > > Key: HDFS-11646 > URL: https://issues.apache.org/jira/browse/HDFS-11646 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HADOOP-11646.patch, HDFS-11646-001.patch, > HDFS-11646-002.patch > > > Add -E option in "ls" to show erasure coding policy of file and directory, > leverage the "number_of_replicas " column. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11647: - Attachment: (was: HADOOP-11647.patch) > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11647-001.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11726) [SPS] : StoragePolicySatisfier should not select same storage type as source and destination in same datanode.
[
https://issues.apache.org/jira/browse/HDFS-11726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-11726:
Resolution: Fixed
Fix Version/s: HDFS-10285
Status: Resolved (was: Patch Available)
Thanks [~umamaheswararao] for the reviews.
Thanks [~surendrasingh] for the patch, I've committed it to branch.
> [SPS] : StoragePolicySatisfier should not select same storage type as source
> and destination in same datanode.
> --
>
> Key: HDFS-11726
> URL: https://issues.apache.org/jira/browse/HDFS-11726
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: HDFS-10285
>
> Attachments: HDFS-11726-HDFS-10285.001.patch,
> HDFS-11726-HDFS-10285.002.patch
>
>
> {code}
> 2017-04-30 16:12:28,569 [BlockMoverTask-0] INFO
> datanode.StoragePolicySatisfyWorker (Worker.java:moveBlock(248)) - Start
> moving block:blk_1073741826_1002 from src:127.0.0.1:41699 to
> destin:127.0.0.1:41699 to satisfy storageType, sourceStoragetype:ARCHIVE and
> destinStoragetype:ARCHIVE
> {code}
> {code}
> 2017-04-30 16:12:28,571 [DataXceiver for client /127.0.0.1:36428 [Replacing
> block BP-1409501412-127.0.1.1-1493548923222:blk_1073741826_1002 from
> 6c7aa66e-a778-43d5-89f6-053d5f6b35bc]] INFO datanode.DataNode
> (DataXceiver.java:replaceBlock(1202)) - opReplaceBlock
> BP-1409501412-127.0.1.1-1493548923222:blk_1073741826_1002 received exception
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
> FinalizedReplica, blk_1073741826_1002, FINALIZED
> getNumBytes() = 1024
> getBytesOnDisk() = 1024
> getVisibleLength()= 1024
> getVolume() =
> /home/sachin/software/hadoop/HDFS-10285/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data7
> getBlockURI() =
> file:/home/sachin/software/hadoop/HDFS-10285/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data7/current/BP-1409501412-127.0.1.1-1493548923222/current/finalized/subdir0/subdir0/blk_1073741826
> already exists on storage ARCHIVE
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-9924) [umbrella] Nonblocking HDFS Access
[ https://issues.apache.org/jira/browse/HDFS-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HDFS-9924: --- Assignee: Duo Zhang (was: Xiaobing Zhou) > [umbrella] Nonblocking HDFS Access > -- > > Key: HDFS-9924 > URL: https://issues.apache.org/jira/browse/HDFS-9924 > Project: Hadoop HDFS > Issue Type: New Feature > Components: fs >Reporter: Tsz Wo Nicholas Sze >Assignee: Duo Zhang > Attachments: AsyncHdfs20160510.pdf, > Async-HDFS-Performance-Report.pdf, HDFS-9924-POC.patch > > > This is an umbrella JIRA for supporting Nonblocking HDFS Access. > Currently, all the API methods are blocking calls -- the caller is blocked > until the method returns. It is very slow if a client makes a large number > of independent calls in a single thread since each call has to wait until the > previous call is finished. It is inefficient if a client needs to create a > large number of threads to invoke the calls. > We propose adding a new API to support nonblocking calls, i.e. the caller is > not blocked. The methods in the new API immediately return a Java Future > object. The return value can be obtained by the usual Future.get() method. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11845) Ozone: Output error when DN handshakes with SCM
[
https://issues.apache.org/jira/browse/HDFS-11845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044131#comment-16044131
]
Hadoop QA commented on HDFS-11845:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
18s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11845 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872215/HDFS-11845-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8ff31995c76c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 2ec6464 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19845/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19845/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: Output error when DN handshakes with SCM
> ---
>
> Key: HDFS-11845
> URL: https://issues.apache.org/jira/browse/HDFS-11845
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: HDFS-11845-HDFS-7240.001.patch
>
>
> When start SCM and DN, there is always an error in SCM log
> {noformat}
> 17/05/17 15:19:59 WARN ipc.Server: IPC Server handler 9 on 9861, call Call#4
> Retry#0
> org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol.getVersion
> from
[jira] [Commented] (HDFS-11185) Ozone: Fix disabled tests
[
https://issues.apache.org/jira/browse/HDFS-11185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044129#comment-16044129
]
Hadoop QA commented on HDFS-11185:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
0s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project: The patch generated 4 new +
0 unchanged - 0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
22s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}157m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.ozone.container.ozoneimpl.TestOzoneContainer |
| | hadoop.ozone.scm.TestXceiverClientManager |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.ozone.scm.node.TestNodeManager |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.ozone.scm.TestContainerSQLCli |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.ozone.container.ozoneimpl.TestRatisManager |
| Timed out junit tests | org.apache.hadoop.cblock.TestLocalBlockCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11185 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872202/HDFS-11185-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 766805f310fa 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11
16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision |
[jira] [Updated] (HDFS-9924) [umbrella] Nonblocking HDFS Access
[ https://issues.apache.org/jira/browse/HDFS-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HDFS-9924: Attachment: HDFS-9924-POC.patch A simple rpc client built on netty. Test mkdirs and getFileInfo in TestAsyncDFSClient. Used to prove that option 3 can work. https://github.com/Apache9/hadoop/commit/aa1623c426ee97398089ba15496651119a1672a9 > [umbrella] Nonblocking HDFS Access > -- > > Key: HDFS-9924 > URL: https://issues.apache.org/jira/browse/HDFS-9924 > Project: Hadoop HDFS > Issue Type: New Feature > Components: fs >Reporter: Tsz Wo Nicholas Sze >Assignee: Xiaobing Zhou > Attachments: AsyncHdfs20160510.pdf, > Async-HDFS-Performance-Report.pdf, HDFS-9924-POC.patch > > > This is an umbrella JIRA for supporting Nonblocking HDFS Access. > Currently, all the API methods are blocking calls -- the caller is blocked > until the method returns. It is very slow if a client makes a large number > of independent calls in a single thread since each call has to wait until the > previous call is finished. It is inefficient if a client needs to create a > large number of threads to invoke the calls. > We propose adding a new API to support nonblocking calls, i.e. the caller is > not blocked. The methods in the new API immediately return a Java Future > object. The return value can be obtained by the usual Future.get() method. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11646) Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable
[
https://issues.apache.org/jira/browse/HDFS-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044122#comment-16044122
]
Hadoop QA commented on HDFS-11646:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HDFS-11646 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-11646 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872216/HDFS-11646-001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19846/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add -E option in 'ls' to list erasure coding policy of each file and
> directory if applicable
>
>
> Key: HDFS-11646
> URL: https://issues.apache.org/jira/browse/HDFS-11646
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Reporter: SammiChen
>Assignee: luhuichun
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HADOOP-11646.patch, HDFS-11646-001.patch
>
>
> Add -E option in "ls" to show erasure coding policy of file and directory,
> leverage the "number_of_replicas " column.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11647: - Attachment: HDFS-11647-001.patch > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HADOOP-11647.patch, HDFS-11647-001.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11646) Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable
[ https://issues.apache.org/jira/browse/HDFS-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11646: - Attachment: HDFS-11646-001.patch > Add -E option in 'ls' to list erasure coding policy of each file and > directory if applicable > > > Key: HDFS-11646 > URL: https://issues.apache.org/jira/browse/HDFS-11646 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HADOOP-11646.patch, HDFS-11646-001.patch > > > Add -E option in "ls" to show erasure coding policy of file and directory, > leverage the "number_of_replicas " column. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
