[jira] [Assigned] (HDFS-11703) [READ] Tests for ProvidedStorageMap
[
https://issues.apache.org/jira/browse/HDFS-11703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas reassigned HDFS-11703:
Assignee: Virajith Jalaparti
> [READ] Tests for ProvidedStorageMap
> ---
>
> Key: HDFS-11703
> URL: https://issues.apache.org/jira/browse/HDFS-11703
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11703-HDFS-9806.001.patch,
> HDFS-11703-HDFS-9806.002.patch
>
>
> Add tests for the {{ProvidedStorageMap}} in the namenode
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11663) [READ] Fix NullPointerException in ProvidedBlocksBuilder
[
https://issues.apache.org/jira/browse/HDFS-11663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas reassigned HDFS-11663:
Assignee: Virajith Jalaparti
> [READ] Fix NullPointerException in ProvidedBlocksBuilder
>
>
> Key: HDFS-11663
> URL: https://issues.apache.org/jira/browse/HDFS-11663
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11663-HDFS-9806.001.patch,
> HDFS-11663-HDFS-9806.002.patch, HDFS-11663-HDFS-9806.003.patch
>
>
> When there are no Datanodes with PROVIDED storage,
> {{ProvidedBlocksBuilder#build}} leads to a {{NullPointerException}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11886) Ozone : improve error handling for putkey operation
[ https://issues.apache.org/jira/browse/HDFS-11886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030676#comment-16030676 ] Weiwei Yang commented on HDFS-11886: Thanks [~vagarychen] for raising up this problem and [~anu] for the design doc. Let me know if I understand this correctly. The proposal adds a *ksm-keys-under-progress.db* in KSM, only if all the steps finish successfully, a key is moved from *ksm-keys-under-progress.db* to *ksm.db*. This introduces more times of writes to disk # put key to inprogress db -> add key # delete key in inprogress db -> commit key 1 # add key to ksm db -> commit key 2 do we really need to persist this? Can we store the state in memory only? Only if all succeed, commit this to *ksm.db*, otherwise dispose it. If KSM crashed before a key is committed, that key won't be written to KSM namespace because that cache after KSM restart will be gone. This is like a write-cache in front of ksm.db. Another question: why we need to return a flag to OzoneHandler to determine if a container needs to be created? I am wondering why we need these additional RPC calls, why not let SCM creates the container on datanodes if necessary and simply return client an open container. Thanks. > Ozone : improve error handling for putkey operation > --- > > Key: HDFS-11886 > URL: https://issues.apache.org/jira/browse/HDFS-11886 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Chen Liang > Attachments: design-notes-putkey.pdf > > > Ozone's putKey operations involve a couple steps: > 1. KSM calls allocateBlock to SCM, writes this info to KSM's local metastore > 2. allocatedBlock gets returned to client, client checks to see if container > needs to be created on datanode, if yes, create the container > 3. writes the data to container. > it is possible that 1 succeeded, but 2 or 3 failed, in this case there will > be an entry in KSM's local metastore, but the key is actually nowhere to be > found. We need to revert 1 is 2 or 3 failed in this case. > To resolve this, we need at least two things to be implemented first. > 1. We need deleteKey() to be added KSM first. > 2. We also need container reports to be implemented first such that SCM can > track whether the container is actually added. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11885) createEncryptionZone should not block on initializing EDEK cache
[
https://issues.apache.org/jira/browse/HDFS-11885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030665#comment-16030665
]
Hadoop QA commented on HDFS-11885:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
37s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 3s{color} | {color:orange} root: The patch generated 5 new + 238 unchanged
- 0 fixed = 243 total (was 238) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
19s{color} | {color:red} hadoop-common-project_hadoop-kms generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
4s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 52s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestAclsEndToEnd |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11885 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12870489/HDFS-11885.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4e9890dad39e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4b4a652 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/19683/artifact/patchprocess/diff-checkstyle-root.txt
|
| javadoc |
[jira] [Commented] (HDFS-11901) Modifier 'static' is redundant for inner enums less
[ https://issues.apache.org/jira/browse/HDFS-11901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030663#comment-16030663 ] ZhangBing Lin commented on HDFS-11901: -- Through the log,test failures are unrelated, > Modifier 'static' is redundant for inner enums less > --- > > Key: HDFS-11901 > URL: https://issues.apache.org/jira/browse/HDFS-11901 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin >Priority: Minor > Attachments: HDFS-11901.001.patch, HDFS-11901.002.patch > > > Java enumeration type is a static constant, implicitly modified with static > final,Modifier 'static' is redundant for inner enums less.So I suggest > deleting the 'static' modifier. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11741) Long running balancer may fail due to expired DataEncryptionKey
[
https://issues.apache.org/jira/browse/HDFS-11741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030649#comment-16030649
]
Hadoop QA commented on HDFS-11741:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 29s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 23 unchanged - 0 fixed = 24 total (was 23) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
40s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 58s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
19s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Possible null pointer dereference of KeyManager.encryptionKey in
org.apache.hadoop.hdfs.server.balancer.KeyManager.newDataEncryptionKey()
Dereferenced at KeyManager.java:KeyManager.encryptionKey in
org.apache.hadoop.hdfs.server.balancer.KeyManager.newDataEncryptionKey()
Dereferenced at KeyManager.java:[line 139] |
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.balancer.TestKeyManager |
| | hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11741 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12870500/HDFS-11741.06.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 564b5eebee40 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4b4a652 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Comment Edited] (HDFS-11774) Ozone:KSM : add deleteVolume
[ https://issues.apache.org/jira/browse/HDFS-11774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030644#comment-16030644 ] Mukul Kumar Singh edited comment on HDFS-11774 at 5/31/17 5:18 AM: --- Thanks for the review [~xyao], Please find my replies as following. 1. MetadataManager#getIterator()/getVolumeRootKey() I would suggest we add Metadata#isVolumeEmpty() instead of raw DB interator for the MetadataManager interface. The DB implementation details can be abstracted from the interface by moving some of the implementation details using DBInterator from VolumeManagerImpl#deleteVolume (Line 296-301) into MetadataManagerImpl#isVolumeEmpty(). done, added a new function in MetadataManagerImpl. 2. ACL/permission check for volume deletion. We should check the owner and ACLs before allowing deleting volumes. I'm OK with just check the owner here or wait for the other ticket on checkVolumeAccess to fix this. As you suggested, I would like to address this as part of check volume access. was (Author: msingh): Thanks for the review [~xyao], Please find my replies as following. 1. MetadataManager#getIterator()/getVolumeRootKey() I would suggest we add Metadata#isVolumeEmpty() instead of raw DB interator for the MetadataManager interface. The DB implementation details can be abstracted from the interface by moving some of the implementation details using DBInterator from VolumeManagerImpl#deleteVolume (Line 296-301) into MetadataManagerImpl#isVolumeEmpty(). .bq done, added a new function in MetadataManagerImpl. 2. ACL/permission check for volume deletion. We should check the owner and ACLs before allowing deleting volumes. I'm OK with just check the owner here or wait for the other ticket on checkVolumeAccess to fix this. .bq As you suggested, I would like to address this as part of check volume access. > Ozone:KSM : add deleteVolume > > > Key: HDFS-11774 > URL: https://issues.apache.org/jira/browse/HDFS-11774 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh > Attachments: HDFS-11774-HDFS-7240.001.patch, > HDFS-11774-HDFS-7240.002.patch > > > Delete a volume if there are no buckets present inside the volume. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11774) Ozone:KSM : add deleteVolume
[ https://issues.apache.org/jira/browse/HDFS-11774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030644#comment-16030644 ] Mukul Kumar Singh commented on HDFS-11774: -- Thanks for the review [~xyao], Please find my replies as following. 1. MetadataManager#getIterator()/getVolumeRootKey() I would suggest we add Metadata#isVolumeEmpty() instead of raw DB interator for the MetadataManager interface. The DB implementation details can be abstracted from the interface by moving some of the implementation details using DBInterator from VolumeManagerImpl#deleteVolume (Line 296-301) into MetadataManagerImpl#isVolumeEmpty(). .bq done, added a new function in MetadataManagerImpl. 2. ACL/permission check for volume deletion. We should check the owner and ACLs before allowing deleting volumes. I'm OK with just check the owner here or wait for the other ticket on checkVolumeAccess to fix this. .bq As you suggested, I would like to address this as part of check volume access. > Ozone:KSM : add deleteVolume > > > Key: HDFS-11774 > URL: https://issues.apache.org/jira/browse/HDFS-11774 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh > Attachments: HDFS-11774-HDFS-7240.001.patch, > HDFS-11774-HDFS-7240.002.patch > > > Delete a volume if there are no buckets present inside the volume. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11774) Ozone:KSM : add deleteVolume
[ https://issues.apache.org/jira/browse/HDFS-11774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-11774: - Attachment: HDFS-11774-HDFS-7240.002.patch > Ozone:KSM : add deleteVolume > > > Key: HDFS-11774 > URL: https://issues.apache.org/jira/browse/HDFS-11774 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh > Attachments: HDFS-11774-HDFS-7240.001.patch, > HDFS-11774-HDFS-7240.002.patch > > > Delete a volume if there are no buckets present inside the volume. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11885) createEncryptionZone should not block on initializing EDEK cache
[
https://issues.apache.org/jira/browse/HDFS-11885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030623#comment-16030623
]
Xiao Chen commented on HDFS-11885:
--
Thanks [~andrew.wang] for reporting the issue and providing a patch. I think
this is a good fix.
[~shahrs87]'s comments addresses a similar but different problem, but also
sounds like a good improvement. Rushabh, would you mind filing a jira and
attach your patch? Thanks.
Back to this jira. It seems the proactive warming up was added by HDFS-7209,
but now that we have EDEKloader thread, it makes sense to do it there. And most
importantly, async.
Some small comments and 1 question. (Also thanks for the refactoring, looks
pretty good!)
Comments:
- I feel the javadocs on those {{@VisibleForTesting}} methods are a bit
redundant.
- This is existing code, {{EDEKCacheLoader.java}} line 112's param name should
be "keyNames", not "edeks"
- precommit will likely catch this: FSN's {{edekCacheLoader}} should be private.
- I think the new test case's name should be warmup *Dont* block create zone.
- Sleeping for nine 9s' are pretty much stalled, but I think {{Long.MAX_VALUE}}
reads better.
Question:
Wanted to get this all sorted out: It seems after this fix there are only 2
places that could end up with blocking call to KMS:
- startFile which eventually {{generateEncryptedKey}}. With Rushabh's patch
this will become retriable exception IIUC.
- createZone which eventually {{ensureKeyIsInitialized}}. Will this also be
covered? It seems to me {{KeyProvider#getMetadata}} is always a synchronized
call, and there's no client-side cache now. If our goal is createEZ should
never block, shouldn't we fix this too?
> createEncryptionZone should not block on initializing EDEK cache
>
>
> Key: HDFS-11885
> URL: https://issues.apache.org/jira/browse/HDFS-11885
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Affects Versions: 2.6.5
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>Priority: Critical
> Attachments: HDFS-11885.001.patch
>
>
> When creating an encryption zone, we call {{ensureKeyIsInitialized}}, which
> calls {{provider.warmUpEncryptedKeys(keyName)}}. This is a blocking call,
> which attempts to fill the key cache up to the low watermark.
> If the KMS is down or slow, this can take a very long time, and cause the
> createZone RPC to fail with a timeout.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11901) Modifier 'static' is redundant for inner enums less
[
https://issues.apache.org/jira/browse/HDFS-11901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030596#comment-16030596
]
Hadoop QA commented on HDFS-11901:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 55s{color} | {color:orange} hadoop-hdfs-project: The patch generated 7 new +
489 unchanged - 31 fixed = 496 total (was 520) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 30s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
50s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
0s{color} | {color:green} hadoop-hdfs-nfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11901 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12870483/HDFS-11901.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0487698943cc 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11
16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4b4a652 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (HDFS-11898) DFSClient#isHedgedReadsEnabled() should be per client flag
[
https://issues.apache.org/jira/browse/HDFS-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030569#comment-16030569
]
John Zhuge commented on HDFS-11898:
---
[~vinayrpet] For HDFS-11900, I decided to keep the pool static and submitted a
simple patch for option #1 because it was the original design decision. It'd
require careful design and review from original developer and reviewers to
reverse the course. It will take a while.
In your patch, I think {{DFSClient#isHedgedReadsEnabled}} can be simplified to
just {{return hedgedReadEnabled;}}. {{initThreadsNumForHedgedReads}} guarantees
HEDGED_READ_THREAD_POOL is not null and getMaximumPoolSize() > 0.
> DFSClient#isHedgedReadsEnabled() should be per client flag
> ---
>
> Key: HDFS-11898
> URL: https://issues.apache.org/jira/browse/HDFS-11898
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-11898-01.patch
>
>
> DFSClient#isHedgedReadsEnabled() returns value based on static
> {{HEDGED_READ_THREAD_POOL}}.
> Hence if any of the client initialized this in JVM, all remaining client
> reads will be going through hedged read itself.
> This flag should be per client value.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11670) [SPS]: Add storagepolicy command to check the status of storage policy Satisfier
[
https://issues.apache.org/jira/browse/HDFS-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030570#comment-16030570
]
Surendra Singh Lilhore commented on HDFS-11670:
---
Hi [~umamaheswararao]\[~rakeshr],
Do you think this is useful ? Can I provide the patch ?
> [SPS]: Add storagepolicy command to check the status of storage policy
> Satisfier
>
>
> Key: HDFS-11670
> URL: https://issues.apache.org/jira/browse/HDFS-11670
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>
> Its good to have one command to check SPS is enabled or not. Based on this
> user can take the decision to run the Mover.
> {noformat}
> hdfs storagepolicies -policysatisfierstatus
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11359) DFSAdmin report command supports displaying maintenance state datanodes
[
https://issues.apache.org/jira/browse/HDFS-11359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030563#comment-16030563
]
Hadoop QA commented on HDFS-11359:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
278 unchanged - 6 fixed = 280 total (was 284) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
19s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 0s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
19s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}107m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeMXBean |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11359 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12870478/HDFS-11359.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc xml |
| uname | Linux 1f8cb6af2173 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
|
[jira] [Updated] (HDFS-11900) Hedged reads thread pool creation not synchronized
[
https://issues.apache.org/jira/browse/HDFS-11900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-11900:
--
Assignee: John Zhuge
Status: Patch Available (was: Open)
> Hedged reads thread pool creation not synchronized
> --
>
> Key: HDFS-11900
> URL: https://issues.apache.org/jira/browse/HDFS-11900
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HDFS-11900.001.patch
>
>
> *Non-static* synchronized method initThreadsNumForHedgedReads can't
> synchronize the access to the *static* class variable HEDGED_READ_THREAD_POOL.
> {code}
> private static ThreadPoolExecutor HEDGED_READ_THREAD_POOL;
> ...
> private synchronized void initThreadsNumForHedgedReads(int num) {
> {code}
> 2 DFS clients may update the same static variable in a race because the lock
> is on each DFS client object, not on the shared DFSClient class object.
> There are 2 possible fixes:
> 1. "Global thread pool": Change initThreadsNumForHedgedReads to static
> 2. "Per-client thread pool": Change HEDGED_READ_THREAD_POOL to non-static
> From the description for property {{dfs.client.hedged.read.threadpool.size}}:
> {quote}
> to a positive number. The threadpool size is how many threads to dedicate
> to the running of these 'hedged', concurrent reads in your client.
> {quote}
> it seems to indicate the thread pool is per DFS client.
> Let's assume we go with #1 "Global thread pool". One DFS client has the
> property set to 10 in its config, while the other client has the property set
> to 5 in its config, what is supposed to the size of the global thread pool?
> 5? 10? Or 15?
> The 2nd fix seems more reasonable to me.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11359) DFSAdmin report command supports displaying maintenance state datanodes
[
https://issues.apache.org/jira/browse/HDFS-11359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030466#comment-16030466
]
Manoj Govindassamy edited comment on HDFS-11359 at 5/31/17 3:10 AM:
[~linyiqun],
Thanks for the quick turnaround and providing the patch revision.
v005 patch looks good to me, +1.
DFSAdmin.java: (nit comment)
Good that you removed the old code duplication and created a new common method
{{printDataNodeReports()}}. You are passing the State's string also as an
argument, which really can be retrieved from {{AdminStates}}. Maybe the
AdminState's values are not are exactly same as the one you wanted?
was (Author: manojg):
[~linyiqun],
Thanks for the quick turnaround and providing the patch revision.
v005 patch: +1, with a nit comment below
DFSAdmin.java:
Good that you removed the old code duplication and created a new common method
{{printDataNodeReports()}}. You are passing the State's string also as an
argument, which really can be retrieved from {{AdminStates}}. Please check if
you can make use of it. Maybe the AdminState's values are not are exactly same
as the one you wanted?
> DFSAdmin report command supports displaying maintenance state datanodes
> ---
>
> Key: HDFS-11359
> URL: https://issues.apache.org/jira/browse/HDFS-11359
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-11359.001.patch, HDFS-11359.002.patch,
> HDFS-11359.003.patch, HDFS-11359.004.patch, HDFS-11359.005.patch
>
>
> The datanode's maintenance state info can be showed in webUI/JMX. But it
> can't be displayed via CLI. This JIRA will improve on this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11900) Hedged reads thread pool creation not synchronized
[
https://issues.apache.org/jira/browse/HDFS-11900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-11900:
--
Attachment: HDFS-11900.001.patch
Patch 001
* Change {{initThreadsNumForHedgedReads}} to static, thus the *synchonized*
static method can serialize the creation of the pool.
Testing Done
* TestPread
> Hedged reads thread pool creation not synchronized
> --
>
> Key: HDFS-11900
> URL: https://issues.apache.org/jira/browse/HDFS-11900
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.8.0
>Reporter: John Zhuge
> Attachments: HDFS-11900.001.patch
>
>
> *Non-static* synchronized method initThreadsNumForHedgedReads can't
> synchronize the access to the *static* class variable HEDGED_READ_THREAD_POOL.
> {code}
> private static ThreadPoolExecutor HEDGED_READ_THREAD_POOL;
> ...
> private synchronized void initThreadsNumForHedgedReads(int num) {
> {code}
> 2 DFS clients may update the same static variable in a race because the lock
> is on each DFS client object, not on the shared DFSClient class object.
> There are 2 possible fixes:
> 1. "Global thread pool": Change initThreadsNumForHedgedReads to static
> 2. "Per-client thread pool": Change HEDGED_READ_THREAD_POOL to non-static
> From the description for property {{dfs.client.hedged.read.threadpool.size}}:
> {quote}
> to a positive number. The threadpool size is how many threads to dedicate
> to the running of these 'hedged', concurrent reads in your client.
> {quote}
> it seems to indicate the thread pool is per DFS client.
> Let's assume we go with #1 "Global thread pool". One DFS client has the
> property set to 10 in its config, while the other client has the property set
> to 5 in its config, what is supposed to the size of the global thread pool?
> 5? 10? Or 15?
> The 2nd fix seems more reasonable to me.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11900) Hedged reads thread pool creation not synchronized
[
https://issues.apache.org/jira/browse/HDFS-11900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-11900:
--
Summary: Hedged reads thread pool creation not synchronized (was:
initThreadsNumForHedgedReads does not synchronize access to the static pool)
> Hedged reads thread pool creation not synchronized
> --
>
> Key: HDFS-11900
> URL: https://issues.apache.org/jira/browse/HDFS-11900
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>
> *Non-static* synchronized method initThreadsNumForHedgedReads can't
> synchronize the access to the *static* class variable HEDGED_READ_THREAD_POOL.
> {code}
> private static ThreadPoolExecutor HEDGED_READ_THREAD_POOL;
> ...
> private synchronized void initThreadsNumForHedgedReads(int num) {
> {code}
> 2 DFS clients may update the same static variable in a race because the lock
> is on each DFS client object, not on the shared DFSClient class object.
> There are 2 possible fixes:
> 1. "Global thread pool": Change initThreadsNumForHedgedReads to static
> 2. "Per-client thread pool": Change HEDGED_READ_THREAD_POOL to non-static
> From the description for property {{dfs.client.hedged.read.threadpool.size}}:
> {quote}
> to a positive number. The threadpool size is how many threads to dedicate
> to the running of these 'hedged', concurrent reads in your client.
> {quote}
> it seems to indicate the thread pool is per DFS client.
> Let's assume we go with #1 "Global thread pool". One DFS client has the
> property set to 10 in its config, while the other client has the property set
> to 5 in its config, what is supposed to the size of the global thread pool?
> 5? 10? Or 15?
> The 2nd fix seems more reasonable to me.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11904) Reuse iip in unprotectedRemoveXAttrs calls
[
https://issues.apache.org/jira/browse/HDFS-11904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030552#comment-16030552
]
Xiao Chen commented on HDFS-11904:
--
Thanks [~liuml07], good idea to convert to a sub-task.
No tests needed since existing coverage should be ok. Failed tests look
unrelated.
Will commit this in 24 hours if no objections.
> Reuse iip in unprotectedRemoveXAttrs calls
> --
>
> Key: HDFS-11904
> URL: https://issues.apache.org/jira/browse/HDFS-11904
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.7.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-11904.01.patch
>
>
> In HDFS-10939, {{unprotectedSetXAttrs}} was optimized to use IIP instead of
> path string.
> This jira is to do the same on {{unprotectedRemoveXAttrs}}.
> No performance test specifically for this is done yet, but it's not hard to
> see a usage pattern of frequent removexattr could induce perf issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11904) Reuse iip in unprotectedRemoveXAttrs calls
[
https://issues.apache.org/jira/browse/HDFS-11904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-11904:
-
Description:
In HDFS-10939, {{unprotectedSetXAttrs}} was optimized to use IIP instead of
path string.
This jira is to do the same on {{unprotectedRemoveXAttrs}}.
No performance test specifically for this is done yet, but it's not hard to see
a usage pattern of frequent removexattr could induce perf issues.
was:
In HDFS-10939, {{unprotectedSetXAttrs}} was optimized to use IIP instead of
path string.
This jira is to do the same on {{unprotectedRemoveXAttrs}}.
No performance test specifically for this is done yet, but it's not hard to see
if a usage pattern of frequent removexattr could induce perf issues.
> Reuse iip in unprotectedRemoveXAttrs calls
> --
>
> Key: HDFS-11904
> URL: https://issues.apache.org/jira/browse/HDFS-11904
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.7.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-11904.01.patch
>
>
> In HDFS-10939, {{unprotectedSetXAttrs}} was optimized to use IIP instead of
> path string.
> This jira is to do the same on {{unprotectedRemoveXAttrs}}.
> No performance test specifically for this is done yet, but it's not hard to
> see a usage pattern of frequent removexattr could induce perf issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11741) Long running balancer may fail due to expired DataEncryptionKey
[
https://issues.apache.org/jira/browse/HDFS-11741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-11741:
-
Attachment: HDFS-11741.06.patch
Posting a rev based on my review (since Wei-Chiu is on leave) per offline chat
with [~yzhangal].
Yongjun, could you please take a look? Thanks much.
> Long running balancer may fail due to expired DataEncryptionKey
> ---
>
> Key: HDFS-11741
> URL: https://issues.apache.org/jira/browse/HDFS-11741
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
> Environment: CDH5.8.2, Kerberos, Data transfer encryption enabled.
> Balancer login using keytab
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: block keys.png, HDFS-11741.001.patch,
> HDFS-11741.002.patch, HDFS-11741.003.patch, HDFS-11741.004.patch,
> HDFS-11741.005.patch, HDFS-11741.06.patch
>
>
> We found a long running balancer may fail despite using keytab, because
> KeyManager returns expired DataEncryptionKey, and it throws the following
> exception:
> {noformat}
> 2017-04-30 05:03:58,661 WARN [pool-1464-thread-10] balancer.Dispatcher
> (Dispatcher.java:dispatch(325)) - Failed to move blk_1067352712_3913241 with
> size=546650 from 10.0.0.134:50010:DISK to 10.0.0.98:50010:DISK through
> 10.0.0.134:50010
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=1005215027) doesn't exist. Current key: 1005215030
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:474)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:299)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:242)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:183)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.dispatch(Dispatcher.java:311)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$2300(Dispatcher.java:182)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$1.run(Dispatcher.java:899)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This bug is similar in nature to HDFS-10609. While balancer KeyManager
> actively synchronizes itself with NameNode w.r.t block keys, it does not
> update DataEncryptionKey accordingly.
> In a specific cluster, with Kerberos ticket life time 10 hours, and default
> block token expiration/life time 10 hours, a long running balancer failed
> after 20~30 hours.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11708) Positional read will fail if replicas moved to different DNs after stream is opened
[ https://issues.apache.org/jira/browse/HDFS-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030521#comment-16030521 ] Vinayakumar B commented on HDFS-11708: -- Yes. HDFS-11898 should be committed first. > Positional read will fail if replicas moved to different DNs after stream is > opened > --- > > Key: HDFS-11708 > URL: https://issues.apache.org/jira/browse/HDFS-11708 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.7.3 >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Critical > Labels: release-blocker > Attachments: HDFS-11708-01.patch, HDFS-11708-02.patch, > HDFS-11708-03.patch, HDFS-11708-04.patch, HDFS-11708-05.patch, > HDFS-11708-HDFS-11898-06.patch > > > Scenario: > 1. File was written to DN1, DN2 with RF=2 > 2. File stream opened to read and kept. Block Locations are [DN1,DN2] > 3. One of the replica (DN2) moved to another datanode (DN3) due to datanode > dead/balancing/etc. > 4. Latest block locations in NameNode will be DN1 and DN3 in the 'same order' > 5. DN1 went down, but not yet detected as dead in NameNode. > 6. Client start reading using positional read api "read(pos, buf[], offset, > length)" -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11383) String duplication in org.apache.hadoop.fs.BlockLocation
[
https://issues.apache.org/jira/browse/HDFS-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030518#comment-16030518
]
Hadoop QA commented on HDFS-11383:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
20s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 19
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 38s{color} | {color:orange} root: The patch generated 5 new + 24 unchanged -
2 fixed = 29 total (was 26) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
20s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11383 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12870474/HDFS-11383.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 92f4be1edf5e 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4b4a652 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/19680/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/19680/artifact/patchprocess/diff-checkstyle-root.txt
|
| Test Results |
[jira] [Updated] (HDFS-5042) Completed files lost after power failure
[
https://issues.apache.org/jira/browse/HDFS-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-5042:
Summary: Completed files lost after power failure (was:
utuvjndvrcdtgddbdvj)
> Completed files lost after power failure
>
>
> Key: HDFS-5042
> URL: https://issues.apache.org/jira/browse/HDFS-5042
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: ext3 on CentOS 5.7 (kernel 2.6.18-274.el5)
>Reporter: Dave Latham
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-5042-01.patch, HDFS-5042-02.patch,
> HDFS-5042-03.patch, HDFS-5042-04.patch, HDFS-5042-05-branch-2.patch,
> HDFS-5042-05.patch, HDFS-5042-branch-2-01.patch, HDFS-5042-branch-2-05.patch
>
>
> We suffered a cluster wide power failure after which HDFS lost data that it
> had acknowledged as closed and complete.
> The client was HBase which compacted a set of HFiles into a new HFile, then
> after closing the file successfully, deleted the previous versions of the
> file. The cluster then lost power, and when brought back up the newly
> created file was marked CORRUPT.
> Based on reading the logs it looks like the replicas were created by the
> DataNodes in the 'blocksBeingWritten' directory. Then when the file was
> closed they were moved to the 'current' directory. After the power cycle
> those replicas were again in the blocksBeingWritten directory of the
> underlying file system (ext3). When those DataNodes reported in to the
> NameNode it deleted those replicas and lost the file.
> Some possible fixes could be having the DataNode fsync the directory(s) after
> moving the block from blocksBeingWritten to current to ensure the rename is
> durable or having the NameNode accept replicas from blocksBeingWritten under
> certain circumstances.
> Log snippets from RS (RegionServer), NN (NameNode), DN (DataNode):
> {noformat}
> RS 2013-06-29 11:16:06,812 DEBUG org.apache.hadoop.hbase.util.FSUtils:
> Creating
> file=hdfs://hm3:9000/hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> with permission=rwxrwxrwx
> NN 2013-06-29 11:16:06,830 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.allocateBlock:
> /hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c.
> blk_1395839728632046111_357084589
> DN 2013-06-29 11:16:06,832 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_1395839728632046111_357084589 src: /10.0.5.237:14327 dest:
> /10.0.5.237:50010
> NN 2013-06-29 11:16:11,370 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.0.6.1:50010 is added to
> blk_1395839728632046111_357084589 size 25418340
> NN 2013-06-29 11:16:11,370 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.0.6.24:50010 is added to
> blk_1395839728632046111_357084589 size 25418340
> NN 2013-06-29 11:16:11,385 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.0.5.237:50010 is added to
> blk_1395839728632046111_357084589 size 25418340
> DN 2013-06-29 11:16:11,385 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
> blk_1395839728632046111_357084589 of size 25418340 from /10.0.5.237:14327
> DN 2013-06-29 11:16:11,385 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 2 for block
> blk_1395839728632046111_357084589 terminating
> NN 2013-06-29 11:16:11,385 INFO org.apache.hadoop.hdfs.StateChange: Removing
> lease on file
> /hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> from client DFSClient_hb_rs_hs745,60020,1372470111932
> NN 2013-06-29 11:16:11,385 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.completeFile: file
> /hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> is closed by DFSClient_hb_rs_hs745,60020,1372470111932
> RS 2013-06-29 11:16:11,393 INFO org.apache.hadoop.hbase.regionserver.Store:
> Renaming compacted file at
> hdfs://hm3:9000/hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> to
> hdfs://hm3:9000/hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/n/6e0cc30af6e64e56ba5a539fdf159c4c
> RS 2013-06-29 11:16:11,505 INFO org.apache.hadoop.hbase.regionserver.Store:
> Completed major compaction of 7 file(s) in n of
> users-6,\x12\xBDp\xA3,1359426311784.b5b0820cde759ae68e333b2f4015bb7e. into
> 6e0cc30af6e64e56ba5a539fdf159c4c, size=24.2m; total size for store is 24.2m
> --- CRASH, RESTART -
> NN 2013-06-29 12:01:19,743 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: addStoredBlock request received for
> blk_1395839728632046111_357084589 on
[jira] [Commented] (HDFS-11904) Reuse iip in unprotectedRemoveXAttrs calls
[
https://issues.apache.org/jira/browse/HDFS-11904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030510#comment-16030510
]
Hadoop QA commented on HDFS-11904:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 1s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}119m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSAdminWithHA |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11904 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12870467/HDFS-11904.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e26ed5cff27c 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11
16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4b4a652 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19679/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19679/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19679/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Reuse iip in unprotectedRemoveXAttrs calls
> --
>
>
[jira] [Commented] (HDFS-11597) Ozone: Add Ratis management API
[ https://issues.apache.org/jira/browse/HDFS-11597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030506#comment-16030506 ] Tsz Wo Nicholas Sze commented on HDFS-11597: > The question I was trying to ask here was – What happens if the SCM restarts > with this change ? Does SCM remember this Ratis cluster information ? Or do > we expect to relearn that info from heartbeats or something ? This patch does not try to solve this problem. All the states regarding to RatisManager currently is in memory but not persist to disk. As you mentioned, do we want to persist the Ratis cluster states in SCM? Or just use heartbeats to relearn them? I think we should persist the Ratis cluster states since they correspond to the open containers (For closed containers, the Ratis cluster is closed). How does SCM current persist other states? We should do the same for RatisManager. > Ozone: Add Ratis management API > --- > > Key: HDFS-11597 > URL: https://issues.apache.org/jira/browse/HDFS-11597 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-11597-HDFS-7240.20170522.patch, > HDFS-11597-HDFS-7240.20170523.patch, HDFS-11597-HDFS-7240.20170524.patch, > HDFS-11597-HDFS-7240.20170528b.patch, HDFS-11597-HDFS-7240.20170528.patch, > HDFS-11597-HDFS-7240.20170529.patch > > > We need APIs to manage Ratis clusters for the following operations: > - create cluster; > - close cluster; > - get members; and > - update members. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11893) Fix TestDFSShell.testMoveWithTargetPortEmpty failure.
[
https://issues.apache.org/jira/browse/HDFS-11893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-11893:
---
Attachment: org.apache.hadoop.hdfs.TestDFSShell-output.txt
org.apache.hadoop.hdfs.TestDFSShell.txt
Attaching logs after running the following command on trunk with your patch
applied:
{{mvn clean test -Dtest=TestDFSShell#testMoveWithTargetPortEmpty}}
> Fix TestDFSShell.testMoveWithTargetPortEmpty failure.
> -
>
> Key: HDFS-11893
> URL: https://issues.apache.org/jira/browse/HDFS-11893
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.7.4
>Reporter: Konstantin Shvachko
>Assignee: Brahma Reddy Battula
> Labels: release-blocker
> Attachments: HDFS-11893-001.patch,
> org.apache.hadoop.hdfs.TestDFSShell-output.txt,
> org.apache.hadoop.hdfs.TestDFSShell.txt
>
>
> {{TestDFSShell.testMoveWithTargetPortEmpty()}} is consistently failing.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11597) Ozone: Add Ratis management API
[
https://issues.apache.org/jira/browse/HDFS-11597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030500#comment-16030500
]
Tsz Wo Nicholas Sze commented on HDFS-11597:
> What is the semantics of this function ? What does it really do ?
{code}
//RaftClient.java
/** Send reinitialize request to the service. */
RaftClientReply reinitialize(RaftPeer[] serversInNewConf, RaftPeerId server)
throws IOException;
{code}
The client will re-initialize the given server with the pees specified in
serversInNewConf.
This was added by RATIS-86. Before RATIS-86, when a RaftServer starts, it is
initialized with a particular cluster and it cannot join another cluster.
After RATIS-86, when a RaftServer starts, it only initializes an RPC server (so
that it can receive commands) but it does not join any cluster. The
reinitialize(..) method is to reinitialize the server so that it will join a
particular cluster.
> Is this documented anywhere ? or should I look at ratis source code ?
Sorry that the javadoc currently is very simple (will add more details) as
shown above. Please read the source code for the moment.
> Ozone: Add Ratis management API
> ---
>
> Key: HDFS-11597
> URL: https://issues.apache.org/jira/browse/HDFS-11597
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Attachments: HDFS-11597-HDFS-7240.20170522.patch,
> HDFS-11597-HDFS-7240.20170523.patch, HDFS-11597-HDFS-7240.20170524.patch,
> HDFS-11597-HDFS-7240.20170528b.patch, HDFS-11597-HDFS-7240.20170528.patch,
> HDFS-11597-HDFS-7240.20170529.patch
>
>
> We need APIs to manage Ratis clusters for the following operations:
> - create cluster;
> - close cluster;
> - get members; and
> - update members.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5042) utuvjndvrcdtgddbdvj
[
https://issues.apache.org/jira/browse/HDFS-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-5042:
-
Summary: utuvjndvrcdtgddbdvj (was: Completed files lost after power
failure)
> utuvjndvrcdtgddbdvj
> ---
>
> Key: HDFS-5042
> URL: https://issues.apache.org/jira/browse/HDFS-5042
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: ext3 on CentOS 5.7 (kernel 2.6.18-274.el5)
>Reporter: Dave Latham
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-5042-01.patch, HDFS-5042-02.patch,
> HDFS-5042-03.patch, HDFS-5042-04.patch, HDFS-5042-05-branch-2.patch,
> HDFS-5042-05.patch, HDFS-5042-branch-2-01.patch, HDFS-5042-branch-2-05.patch
>
>
> We suffered a cluster wide power failure after which HDFS lost data that it
> had acknowledged as closed and complete.
> The client was HBase which compacted a set of HFiles into a new HFile, then
> after closing the file successfully, deleted the previous versions of the
> file. The cluster then lost power, and when brought back up the newly
> created file was marked CORRUPT.
> Based on reading the logs it looks like the replicas were created by the
> DataNodes in the 'blocksBeingWritten' directory. Then when the file was
> closed they were moved to the 'current' directory. After the power cycle
> those replicas were again in the blocksBeingWritten directory of the
> underlying file system (ext3). When those DataNodes reported in to the
> NameNode it deleted those replicas and lost the file.
> Some possible fixes could be having the DataNode fsync the directory(s) after
> moving the block from blocksBeingWritten to current to ensure the rename is
> durable or having the NameNode accept replicas from blocksBeingWritten under
> certain circumstances.
> Log snippets from RS (RegionServer), NN (NameNode), DN (DataNode):
> {noformat}
> RS 2013-06-29 11:16:06,812 DEBUG org.apache.hadoop.hbase.util.FSUtils:
> Creating
> file=hdfs://hm3:9000/hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> with permission=rwxrwxrwx
> NN 2013-06-29 11:16:06,830 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.allocateBlock:
> /hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c.
> blk_1395839728632046111_357084589
> DN 2013-06-29 11:16:06,832 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_1395839728632046111_357084589 src: /10.0.5.237:14327 dest:
> /10.0.5.237:50010
> NN 2013-06-29 11:16:11,370 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.0.6.1:50010 is added to
> blk_1395839728632046111_357084589 size 25418340
> NN 2013-06-29 11:16:11,370 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.0.6.24:50010 is added to
> blk_1395839728632046111_357084589 size 25418340
> NN 2013-06-29 11:16:11,385 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.0.5.237:50010 is added to
> blk_1395839728632046111_357084589 size 25418340
> DN 2013-06-29 11:16:11,385 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
> blk_1395839728632046111_357084589 of size 25418340 from /10.0.5.237:14327
> DN 2013-06-29 11:16:11,385 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 2 for block
> blk_1395839728632046111_357084589 terminating
> NN 2013-06-29 11:16:11,385 INFO org.apache.hadoop.hdfs.StateChange: Removing
> lease on file
> /hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> from client DFSClient_hb_rs_hs745,60020,1372470111932
> NN 2013-06-29 11:16:11,385 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.completeFile: file
> /hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> is closed by DFSClient_hb_rs_hs745,60020,1372470111932
> RS 2013-06-29 11:16:11,393 INFO org.apache.hadoop.hbase.regionserver.Store:
> Renaming compacted file at
> hdfs://hm3:9000/hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> to
> hdfs://hm3:9000/hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/n/6e0cc30af6e64e56ba5a539fdf159c4c
> RS 2013-06-29 11:16:11,505 INFO org.apache.hadoop.hbase.regionserver.Store:
> Completed major compaction of 7 file(s) in n of
> users-6,\x12\xBDp\xA3,1359426311784.b5b0820cde759ae68e333b2f4015bb7e. into
> 6e0cc30af6e64e56ba5a539fdf159c4c, size=24.2m; total size for store is 24.2m
> --- CRASH, RESTART -
> NN 2013-06-29 12:01:19,743 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: addStoredBlock request received for
> blk_1395839728632046111_357084589 on 10.0.6.1:50010 size 21978112 but was
>
[jira] [Commented] (HDFS-11885) createEncryptionZone should not block on initializing EDEK cache
[
https://issues.apache.org/jira/browse/HDFS-11885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030491#comment-16030491
]
Andrew Wang commented on HDFS-11885:
[~xiaochen] could you review, since I think you worked on this warm-up code
originally?
> createEncryptionZone should not block on initializing EDEK cache
>
>
> Key: HDFS-11885
> URL: https://issues.apache.org/jira/browse/HDFS-11885
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Affects Versions: 2.6.5
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>Priority: Critical
> Attachments: HDFS-11885.001.patch
>
>
> When creating an encryption zone, we call {{ensureKeyIsInitialized}}, which
> calls {{provider.warmUpEncryptedKeys(keyName)}}. This is a blocking call,
> which attempts to fill the key cache up to the low watermark.
> If the KMS is down or slow, this can take a very long time, and cause the
> createZone RPC to fail with a timeout.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11885) createEncryptionZone should not block on initializing EDEK cache
[
https://issues.apache.org/jira/browse/HDFS-11885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-11885:
---
Attachment: HDFS-11885.001.patch
Here's a patch which breaks out the EDEKCacheLoader into its own class, and
uses it to run the EDEK warm up in the background on createZone.
Rushabh, thanks for the offer. createZone still queries the KMS synchronously
in {{ensureKeyIsInitialized}} to see if the key exists. Your proposed patch
would still help with that. This JIRA is just to help with slow generation of
EDEKs.
> createEncryptionZone should not block on initializing EDEK cache
>
>
> Key: HDFS-11885
> URL: https://issues.apache.org/jira/browse/HDFS-11885
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Affects Versions: 2.6.5
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>Priority: Critical
> Attachments: HDFS-11885.001.patch
>
>
> When creating an encryption zone, we call {{ensureKeyIsInitialized}}, which
> calls {{provider.warmUpEncryptedKeys(keyName)}}. This is a blocking call,
> which attempts to fill the key cache up to the low watermark.
> If the KMS is down or slow, this can take a very long time, and cause the
> createZone RPC to fail with a timeout.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11885) createEncryptionZone should not block on initializing EDEK cache
[
https://issues.apache.org/jira/browse/HDFS-11885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-11885:
---
Status: Patch Available (was: Open)
> createEncryptionZone should not block on initializing EDEK cache
>
>
> Key: HDFS-11885
> URL: https://issues.apache.org/jira/browse/HDFS-11885
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Affects Versions: 2.6.5
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>Priority: Critical
> Attachments: HDFS-11885.001.patch
>
>
> When creating an encryption zone, we call {{ensureKeyIsInitialized}}, which
> calls {{provider.warmUpEncryptedKeys(keyName)}}. This is a blocking call,
> which attempts to fill the key cache up to the low watermark.
> If the KMS is down or slow, this can take a very long time, and cause the
> createZone RPC to fail with a timeout.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11708) Positional read will fail if replicas moved to different DNs after stream is opened
[
https://issues.apache.org/jira/browse/HDFS-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030486#comment-16030486
]
Konstantin Shvachko commented on HDFS-11708:
I like the idea of returning updated block in {{DNAddrPair}} and refreshing it
after every {{chooseTarget()}} call.
Looks like you submitted two patches for both jiras here. So what do we do?
Should we first commit HDFS-11898?
> Positional read will fail if replicas moved to different DNs after stream is
> opened
> ---
>
> Key: HDFS-11708
> URL: https://issues.apache.org/jira/browse/HDFS-11708
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.3
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-11708-01.patch, HDFS-11708-02.patch,
> HDFS-11708-03.patch, HDFS-11708-04.patch, HDFS-11708-05.patch,
> HDFS-11708-HDFS-11898-06.patch
>
>
> Scenario:
> 1. File was written to DN1, DN2 with RF=2
> 2. File stream opened to read and kept. Block Locations are [DN1,DN2]
> 3. One of the replica (DN2) moved to another datanode (DN3) due to datanode
> dead/balancing/etc.
> 4. Latest block locations in NameNode will be DN1 and DN3 in the 'same order'
> 5. DN1 went down, but not yet detected as dead in NameNode.
> 6. Client start reading using positional read api "read(pos, buf[], offset,
> length)"
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11726) [SPS] : StoragePolicySatisfier should not select same storage type as source and destination in same datanode.
[
https://issues.apache.org/jira/browse/HDFS-11726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030471#comment-16030471
]
Uma Maheswara Rao G commented on HDFS-11726:
ah got your scenario Surendra. I will review your patch in some time. Thanks
for the fix.
> [SPS] : StoragePolicySatisfier should not select same storage type as source
> and destination in same datanode.
> --
>
> Key: HDFS-11726
> URL: https://issues.apache.org/jira/browse/HDFS-11726
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-11726-HDFS-10285.001.patch
>
>
> {code}
> 2017-04-30 16:12:28,569 [BlockMoverTask-0] INFO
> datanode.StoragePolicySatisfyWorker (Worker.java:moveBlock(248)) - Start
> moving block:blk_1073741826_1002 from src:127.0.0.1:41699 to
> destin:127.0.0.1:41699 to satisfy storageType, sourceStoragetype:ARCHIVE and
> destinStoragetype:ARCHIVE
> {code}
> {code}
> 2017-04-30 16:12:28,571 [DataXceiver for client /127.0.0.1:36428 [Replacing
> block BP-1409501412-127.0.1.1-1493548923222:blk_1073741826_1002 from
> 6c7aa66e-a778-43d5-89f6-053d5f6b35bc]] INFO datanode.DataNode
> (DataXceiver.java:replaceBlock(1202)) - opReplaceBlock
> BP-1409501412-127.0.1.1-1493548923222:blk_1073741826_1002 received exception
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
> FinalizedReplica, blk_1073741826_1002, FINALIZED
> getNumBytes() = 1024
> getBytesOnDisk() = 1024
> getVisibleLength()= 1024
> getVolume() =
> /home/sachin/software/hadoop/HDFS-10285/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data7
> getBlockURI() =
> file:/home/sachin/software/hadoop/HDFS-10285/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data7/current/BP-1409501412-127.0.1.1-1493548923222/current/finalized/subdir0/subdir0/blk_1073741826
> already exists on storage ARCHIVE
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11359) DFSAdmin report command supports displaying maintenance state datanodes
[
https://issues.apache.org/jira/browse/HDFS-11359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030466#comment-16030466
]
Manoj Govindassamy commented on HDFS-11359:
---
[~linyiqun],
Thanks for the quick turnaround and providing the patch revision.
v005 patch: +1, with a nit comment below
DFSAdmin.java:
Good that you removed the old code duplication and created a new common method
{{printDataNodeReports()}}. You are passing the State's string also as an
argument, which really can be retrieved from {{AdminStates}}. Please check if
you can make use of it. Maybe the AdminState's values are not are exactly same
as the one you wanted?
> DFSAdmin report command supports displaying maintenance state datanodes
> ---
>
> Key: HDFS-11359
> URL: https://issues.apache.org/jira/browse/HDFS-11359
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-11359.001.patch, HDFS-11359.002.patch,
> HDFS-11359.003.patch, HDFS-11359.004.patch, HDFS-11359.005.patch
>
>
> The datanode's maintenance state info can be showed in webUI/JMX. But it
> can't be displayed via CLI. This JIRA will improve on this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11883) [SPS] : Handle NPE in BlockStorageMovementTracker when dropSPSWork() called
[
https://issues.apache.org/jira/browse/HDFS-11883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-11883:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-10285
Status: Resolved (was: Patch Available)
I have just pushed this to branch. Thanks [~surendrasingh]
> [SPS] : Handle NPE in BlockStorageMovementTracker when dropSPSWork() called
> ---
>
> Key: HDFS-11883
> URL: https://issues.apache.org/jira/browse/HDFS-11883
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: HDFS-10285
>
> Attachments: HDFS-11883.001.patch, HDFS-11883-HDFS-10285.001.patch,
> HDFS-11883-HDFS-10285.002.patch
>
>
> {noformat}
> Exception in thread "BlockStorageMovementTracker"
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.BlockStorageMovementTracker.run(BlockStorageMovementTracker.java:91)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11901) Modifier 'static' is redundant for inner enums less
[ https://issues.apache.org/jira/browse/HDFS-11901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030456#comment-16030456 ] ZhangBing Lin commented on HDFS-11901: -- Through the log,I think it would't cause these problem! > Modifier 'static' is redundant for inner enums less > --- > > Key: HDFS-11901 > URL: https://issues.apache.org/jira/browse/HDFS-11901 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin >Priority: Minor > Attachments: HDFS-11901.001.patch, HDFS-11901.002.patch > > > Java enumeration type is a static constant, implicitly modified with static > final,Modifier 'static' is redundant for inner enums less.So I suggest > deleting the 'static' modifier. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11901) Modifier 'static' is redundant for inner enums less
[ https://issues.apache.org/jira/browse/HDFS-11901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030450#comment-16030450 ] ZhangBing Lin commented on HDFS-11901: -- I re-submit a new patch and will trigger the jenkins again > Modifier 'static' is redundant for inner enums less > --- > > Key: HDFS-11901 > URL: https://issues.apache.org/jira/browse/HDFS-11901 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin >Priority: Minor > Attachments: HDFS-11901.001.patch, HDFS-11901.002.patch > > > Java enumeration type is a static constant, implicitly modified with static > final,Modifier 'static' is redundant for inner enums less.So I suggest > deleting the 'static' modifier. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11901) Modifier 'static' is redundant for inner enums less
[ https://issues.apache.org/jira/browse/HDFS-11901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhangBing Lin updated HDFS-11901: - Status: Patch Available (was: Open) > Modifier 'static' is redundant for inner enums less > --- > > Key: HDFS-11901 > URL: https://issues.apache.org/jira/browse/HDFS-11901 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin >Priority: Minor > Attachments: HDFS-11901.001.patch, HDFS-11901.002.patch > > > Java enumeration type is a static constant, implicitly modified with static > final,Modifier 'static' is redundant for inner enums less.So I suggest > deleting the 'static' modifier. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11901) Modifier 'static' is redundant for inner enums less
[ https://issues.apache.org/jira/browse/HDFS-11901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhangBing Lin updated HDFS-11901: - Attachment: HDFS-11901.002.patch > Modifier 'static' is redundant for inner enums less > --- > > Key: HDFS-11901 > URL: https://issues.apache.org/jira/browse/HDFS-11901 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin >Priority: Minor > Attachments: HDFS-11901.001.patch, HDFS-11901.002.patch > > > Java enumeration type is a static constant, implicitly modified with static > final,Modifier 'static' is redundant for inner enums less.So I suggest > deleting the 'static' modifier. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11901) Modifier 'static' is redundant for inner enums less
[ https://issues.apache.org/jira/browse/HDFS-11901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhangBing Lin updated HDFS-11901: - Status: Open (was: Patch Available) > Modifier 'static' is redundant for inner enums less > --- > > Key: HDFS-11901 > URL: https://issues.apache.org/jira/browse/HDFS-11901 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin >Priority: Minor > Attachments: HDFS-11901.001.patch, HDFS-11901.002.patch > > > Java enumeration type is a static constant, implicitly modified with static > final,Modifier 'static' is redundant for inner enums less.So I suggest > deleting the 'static' modifier. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11903) Ozone: Cleaning up local storage when closing MiniOzoneCluster
[
https://issues.apache.org/jira/browse/HDFS-11903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030444#comment-16030444
]
Hadoop QA commented on HDFS-11903:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
0s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 54s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}100m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11903 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12870453/HDFS-11903-HDFS-7240.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 20f2026c9b75 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 122d660 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/19678/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19678/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19678/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-11359) DFSAdmin report command supports displaying maintenance state datanodes
[ https://issues.apache.org/jira/browse/HDFS-11359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030431#comment-16030431 ] Mingliang Liu commented on HDFS-11359: -- Thanks, I'm still +1 on this. > DFSAdmin report command supports displaying maintenance state datanodes > --- > > Key: HDFS-11359 > URL: https://issues.apache.org/jira/browse/HDFS-11359 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Minor > Attachments: HDFS-11359.001.patch, HDFS-11359.002.patch, > HDFS-11359.003.patch, HDFS-11359.004.patch, HDFS-11359.005.patch > > > The datanode's maintenance state info can be showed in webUI/JMX. But it > can't be displayed via CLI. This JIRA will improve on this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11840) Log HDFS Mover exception message of exit to its own log
[ https://issues.apache.org/jira/browse/HDFS-11840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11840: --- Fix Version/s: (was: 3.0.0-alpha2) > Log HDFS Mover exception message of exit to its own log > --- > > Key: HDFS-11840 > URL: https://issues.apache.org/jira/browse/HDFS-11840 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 3.0.0-alpha2 >Reporter: LiXin Ge >Assignee: LiXin Ge >Priority: Minor > Attachments: HDFS-11840.patch, HDFS-11840.update.patch > > > Currently, the exception message of why mover exit is logged to stderr. > It is hard to figure out why Mover was aborted as we may lose the console > message,but it would be much better if we also log this to Mover log. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11359) DFSAdmin report command supports displaying maintenance state datanodes
[ https://issues.apache.org/jira/browse/HDFS-11359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11359: - Attachment: HDFS-11359.005.patch Thanks [~liuml07] for the review! All the comments make sense to me. Attach the new patch. Thanks for taking a look. > DFSAdmin report command supports displaying maintenance state datanodes > --- > > Key: HDFS-11359 > URL: https://issues.apache.org/jira/browse/HDFS-11359 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Minor > Attachments: HDFS-11359.001.patch, HDFS-11359.002.patch, > HDFS-11359.003.patch, HDFS-11359.004.patch, HDFS-11359.005.patch > > > The datanode's maintenance state info can be showed in webUI/JMX. But it > can't be displayed via CLI. This JIRA will improve on this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11840) Log HDFS Mover exception message of exit to its own log
[ https://issues.apache.org/jira/browse/HDFS-11840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11840: --- Target Version/s: 3.0.0-alpha4 (was: 3.0.0-alpha2) > Log HDFS Mover exception message of exit to its own log > --- > > Key: HDFS-11840 > URL: https://issues.apache.org/jira/browse/HDFS-11840 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 3.0.0-alpha2 >Reporter: LiXin Ge >Assignee: LiXin Ge >Priority: Minor > Attachments: HDFS-11840.patch, HDFS-11840.update.patch > > > Currently, the exception message of why mover exit is logged to stderr. > It is hard to figure out why Mover was aborted as we may lose the console > message,but it would be much better if we also log this to Mover log. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11902) [READ] Merge BlockFormatProvider and TextFileRegionProvider into one.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030413#comment-16030413
]
Hadoop QA commented on HDFS-11902:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
50s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
8s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
13s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s{color} | {color:green} HDFS-9806 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
34s{color} | {color:red} hadoop-tools/hadoop-fs2img in HDFS-9806 has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} HDFS-9806 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 16s{color} | {color:orange} root: The patch generated 1 new + 432 unchanged
- 3 fixed = 433 total (was 435) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 19s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
49s{color} | {color:green} hadoop-fs2img in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}150m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11902 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12870448/HDFS-11902-HDFS-9806.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux edb2f03b941f 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-9806 / 5d021f3 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
[jira] [Commented] (HDFS-6984) In Hadoop 3, make FileStatus serialize itself via protobuf
[
https://issues.apache.org/jira/browse/HDFS-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030394#comment-16030394
]
Chris Douglas commented on HDFS-6984:
-
[~andrew.wang] do you have cycles to review this?
> In Hadoop 3, make FileStatus serialize itself via protobuf
> --
>
> Key: HDFS-6984
> URL: https://issues.apache.org/jira/browse/HDFS-6984
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Labels: BB2015-05-TBR
> Attachments: HDFS-6984.001.patch, HDFS-6984.002.patch,
> HDFS-6984.003.patch, HDFS-6984.004.patch, HDFS-6984.005.patch,
> HDFS-6984.006.patch, HDFS-6984.007.patch, HDFS-6984.008.patch,
> HDFS-6984.009.patch, HDFS-6984.010.patch, HDFS-6984.011.patch,
> HDFS-6984.012.patch, HDFS-6984.013.patch, HDFS-6984.nowritable.patch
>
>
> FileStatus was a Writable in Hadoop 2 and earlier. Originally, we used this
> to serialize it and send it over the wire. But in Hadoop 2 and later, we
> have the protobuf {{HdfsFileStatusProto}} which serves to serialize this
> information. The protobuf form is preferable, since it allows us to add new
> fields in a backwards-compatible way. Another issue is that already a lot of
> subclasses of FileStatus don't override the Writable methods of the
> superclass, breaking the interface contract that read(status.write) should be
> equal to the original status.
> In Hadoop 3, we should just make FileStatus serialize itself via protobuf so
> that we don't have to deal with these issues. It's probably too late to do
> this in Hadoop 2, since user code may be relying on the existing FileStatus
> serialization there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11383) String duplication in org.apache.hadoop.fs.BlockLocation
[
https://issues.apache.org/jira/browse/HDFS-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-11383:
--
Status: Patch Available (was: In Progress)
> String duplication in org.apache.hadoop.fs.BlockLocation
>
>
> Key: HDFS-11383
> URL: https://issues.apache.org/jira/browse/HDFS-11383
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HDFS-11383.01.patch, HDFS-11383.02.patch,
> HDFS-11383.03.patch, hs2-crash-2.txt
>
>
> I am working on Hive performance, investigating the problem of high memory
> pressure when (a) a table consists of a high number (thousands) of partitions
> and (b) multiple queries run against it concurrently. It turns out that a lot
> of memory is wasted due to data duplication. One source of duplicate strings
> is class org.apache.hadoop.fs.BlockLocation. Its fields such as storageIds,
> topologyPaths, hosts, names, may collectively use up to 6% of memory in my
> benchmark, causing (together with other problematic classes) a huge memory
> spike. Of these 6% of memory taken by BlockLocation strings, more than 5% are
> wasted due to duplication.
> I think we need to add calls to String.intern() in the BlockLocation
> constructor, like:
> {code}
> this.hosts = internStringsInArray(hosts);
> ...
> private void internStringsInArray(String[] sar) {
> for (int i = 0; i < sar.length; i++) {
> sar[i] = sar[i].intern();
> }
> }
> {code}
> String.intern() performs very well starting from JDK 7. I've found some
> articles explaining the progress that was made by the HotSpot JVM developers
> in this area, verified that with benchmarks myself, and finally added quite a
> bit of interning to one of the Cloudera products without any issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11383) String duplication in org.apache.hadoop.fs.BlockLocation
[
https://issues.apache.org/jira/browse/HDFS-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-11383:
--
Attachment: HDFS-11383.03.patch
> String duplication in org.apache.hadoop.fs.BlockLocation
>
>
> Key: HDFS-11383
> URL: https://issues.apache.org/jira/browse/HDFS-11383
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HDFS-11383.01.patch, HDFS-11383.02.patch,
> HDFS-11383.03.patch, hs2-crash-2.txt
>
>
> I am working on Hive performance, investigating the problem of high memory
> pressure when (a) a table consists of a high number (thousands) of partitions
> and (b) multiple queries run against it concurrently. It turns out that a lot
> of memory is wasted due to data duplication. One source of duplicate strings
> is class org.apache.hadoop.fs.BlockLocation. Its fields such as storageIds,
> topologyPaths, hosts, names, may collectively use up to 6% of memory in my
> benchmark, causing (together with other problematic classes) a huge memory
> spike. Of these 6% of memory taken by BlockLocation strings, more than 5% are
> wasted due to duplication.
> I think we need to add calls to String.intern() in the BlockLocation
> constructor, like:
> {code}
> this.hosts = internStringsInArray(hosts);
> ...
> private void internStringsInArray(String[] sar) {
> for (int i = 0; i < sar.length; i++) {
> sar[i] = sar[i].intern();
> }
> }
> {code}
> String.intern() performs very well starting from JDK 7. I've found some
> articles explaining the progress that was made by the HotSpot JVM developers
> in this area, verified that with benchmarks myself, and finally added quite a
> bit of interning to one of the Cloudera products without any issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11904) Reuse iip in unprotectedRemoveXAttrs calls
[
https://issues.apache.org/jira/browse/HDFS-11904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030381#comment-16030381
]
Mingliang Liu commented on HDFS-11904:
--
+1
I converted this to a sub-task of [HDFS-10616].
> Reuse iip in unprotectedRemoveXAttrs calls
> --
>
> Key: HDFS-11904
> URL: https://issues.apache.org/jira/browse/HDFS-11904
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.7.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-11904.01.patch
>
>
> In HDFS-10939, {{unprotectedSetXAttrs}} was optimized to use IIP instead of
> path string.
> This jira is to do the same on {{unprotectedRemoveXAttrs}}.
> No performance test specifically for this is done yet, but it's not hard to
> see if a usage pattern of frequent removexattr could induce perf issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11904) Reuse iip in unprotectedRemoveXAttrs calls
[
https://issues.apache.org/jira/browse/HDFS-11904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-11904:
-
Issue Type: Sub-task (was: Improvement)
Parent: HDFS-10616
> Reuse iip in unprotectedRemoveXAttrs calls
> --
>
> Key: HDFS-11904
> URL: https://issues.apache.org/jira/browse/HDFS-11904
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.7.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-11904.01.patch
>
>
> In HDFS-10939, {{unprotectedSetXAttrs}} was optimized to use IIP instead of
> path string.
> This jira is to do the same on {{unprotectedRemoveXAttrs}}.
> No performance test specifically for this is done yet, but it's not hard to
> see if a usage pattern of frequent removexattr could induce perf issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11904) Reuse iip in unprotectedRemoveXAttrs calls
[
https://issues.apache.org/jira/browse/HDFS-11904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-11904:
-
Description:
In HDFS-10939, {{unprotectedSetXAttrs}} was optimized to use IIP instead of
path string.
This jira is to do the same on {{unprotectedRemoveXAttrs}}.
No performance test specifically for this is done yet, but it's not hard to see
if a usage pattern of frequent removexattr could induce perf issues.
was:
In HDFS-10939, {{unprotectedSetXAttrs}} was optimized to use IIP instead of
path string.
This jira is to do the same on {{unprotectedRemoveXAttrs}}.
No performance test specifically for this is done, but since the optimization
is trivial to do I think we should do it to save future effort.
> Reuse iip in unprotectedRemoveXAttrs calls
> --
>
> Key: HDFS-11904
> URL: https://issues.apache.org/jira/browse/HDFS-11904
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-11904.01.patch
>
>
> In HDFS-10939, {{unprotectedSetXAttrs}} was optimized to use IIP instead of
> path string.
> This jira is to do the same on {{unprotectedRemoveXAttrs}}.
> No performance test specifically for this is done yet, but it's not hard to
> see if a usage pattern of frequent removexattr could induce perf issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11904) Reuse iip in unprotectedRemoveXAttrs calls
[
https://issues.apache.org/jira/browse/HDFS-11904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-11904:
-
Attachment: HDFS-11904.01.patch
> Reuse iip in unprotectedRemoveXAttrs calls
> --
>
> Key: HDFS-11904
> URL: https://issues.apache.org/jira/browse/HDFS-11904
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-11904.01.patch
>
>
> In HDFS-10939, {{unprotectedSetXAttrs}} was optimized to use IIP instead of
> path string.
> This jira is to do the same on {{unprotectedRemoveXAttrs}}.
> No performance test specifically for this is done, but since the optimization
> is trivial to do I think we should do it to save future effort.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11904) Reuse iip in unprotectedRemoveXAttrs calls
[
https://issues.apache.org/jira/browse/HDFS-11904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-11904:
-
Status: Patch Available (was: Open)
> Reuse iip in unprotectedRemoveXAttrs calls
> --
>
> Key: HDFS-11904
> URL: https://issues.apache.org/jira/browse/HDFS-11904
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-11904.01.patch
>
>
> In HDFS-10939, {{unprotectedSetXAttrs}} was optimized to use IIP instead of
> path string.
> This jira is to do the same on {{unprotectedRemoveXAttrs}}.
> No performance test specifically for this is done, but since the optimization
> is trivial to do I think we should do it to save future effort.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11904) Reuse iip in unprotectedRemoveXAttrs calls
Xiao Chen created HDFS-11904:
Summary: Reuse iip in unprotectedRemoveXAttrs calls
Key: HDFS-11904
URL: https://issues.apache.org/jira/browse/HDFS-11904
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.7.0
Reporter: Xiao Chen
Assignee: Xiao Chen
In HDFS-10939, {{unprotectedSetXAttrs}} was optimized to use IIP instead of
path string.
This jira is to do the same on {{unprotectedRemoveXAttrs}}.
No performance test specifically for this is done, but since the optimization
is trivial to do I think we should do it to save future effort.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11903) Ozone: Cleaning up local storage when closing MiniOzoneCluster
[
https://issues.apache.org/jira/browse/HDFS-11903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030302#comment-16030302
]
Mingliang Liu commented on HDFS-11903:
--
Thanks [~anu] for review and for the suggestion. Yes I noticed that I can set
the {{OzoneConfigKeys.OZONE_LOCALSTORAGE_ROOT}} in OzoneConfig and build a new
MiniOzoneCluster with that config object.
One possible problem is that, the {{OzoneMetadataManager}} is singleton, so
only the first OzoneConfig is used to check the local
{{OzoneMetadataManager::storageRoot}} so the {{/metadata.db}} and {{/user.db}}
will be shared. I currently don't need to create the two {{OzoneMiniCluster}}
now, but I assume that will be useful someday, say running distcp between them.
So overall, current test will fail with duplicate volume/bucket exception:
{code}
@Test
public void testCloseWillCleanFiles() throws Exception {
final String vName = "test-close-will-clean-files-volume";
final String bName = "test-close-will-clean-files-bucket";
try (MiniOzoneCluster c1 = new MiniOzoneCluster.Builder(new
OzoneConfiguration())
.numDataNodes(1)
.setHandlerType("local")
.build()) {
OzoneVolume v = c1.createOzoneClient().createVolume(vName, "L", "1TB");
v.createBucket(bName);
}
try (MiniOzoneCluster c2 = new MiniOzoneCluster.Builder(new
OzoneConfiguration())
.numDataNodes(1)
.setHandlerType("local")
.build()) {
OzoneVolume v = c2.createOzoneClient().createVolume(vName, "L", "1TB");
v.createBucket(bName);
}
}
{code}
And it should fail even when we set different
{{OzoneConfigKeys.OZONE_LOCALSTORAGE_ROOT}} I assume (not verified).
> Ozone: Cleaning up local storage when closing MiniOzoneCluster
> --
>
> Key: HDFS-11903
> URL: https://issues.apache.org/jira/browse/HDFS-11903
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-11903-HDFS-7240.000.patch
>
>
> Currently we don't delete the local storage when closing
> {{MiniOzoneCluster}}, which will cause creating volume with the same name to
> fail, even in a new JVM process. Let's delete the local storage files:
> {{/tmp/ozone/metadata.db}} and {{/tmp/ozone/user.db}} when closing.
> Please note this is not a complete solution. As the {{OzoneMetadataManager}}
> currently is a singleton, we are not able to create two independent
> MiniOzoneCluster in a single JVM. Data maybe shared unexpectedly. Let's
> address that in another JIRA, if needed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11359) DFSAdmin report command supports displaying maintenance state datanodes
[
https://issues.apache.org/jira/browse/HDFS-11359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030292#comment-16030292
]
Mingliang Liu commented on HDFS-11359:
--
+1
Minor comments:
# {{printDataNodeReports()}} can either be static, or use {{getFs()}} to get
the FS object to avoid dfs parameter.
# The initial {{putNodeToService()}} seems unnecessary as the node status
should be normal I think.
{code}
1154for (DataNode dn : getCluster().getDataNodes()) {
1155 hosts.add(dn.getDisplayName());
1156 putNodeInService(0, dn.getDatanodeUuid());
1157}
{code}
Can we change this part to {{ToolRunner.run(dfsAdmin, new String[] \{"-report",
"-inmaintenance", "-enteringmaintenance"});}} and assert out does not contain
either of the two DNs?
# {{List hosts = new ArrayList<>();}} in test, this is never used?
# simpler assert statements:
{code}
assertThat(out.toString(),
is(allOf(containsString("Entering maintenance datanodes (1):"),
containsString(nodes[0].getXferAddr();
assertTrue(!out.toString().contains(nodes[1].getXferAddr()));
{code}
to
{code}
assertThat(out.toString(), is(allOf(
containsString("Entering maintenance datanodes (1):"),
containsString(nodes[0].getXferAddr()),
not(containsString(nodes[1].getXferAddr());
{code}
The other place is similar.
# To clear the old out/err,
{code}
// reset stream
out = new ByteArrayOutputStream();
err = new ByteArrayOutputStream();
System.setOut(new PrintStream(out));
System.setErr(new PrintStream(err));
{code}
to
{code}
// reset stream
out.reset();
err.reset();
{code}
> DFSAdmin report command supports displaying maintenance state datanodes
> ---
>
> Key: HDFS-11359
> URL: https://issues.apache.org/jira/browse/HDFS-11359
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-11359.001.patch, HDFS-11359.002.patch,
> HDFS-11359.003.patch, HDFS-11359.004.patch
>
>
> The datanode's maintenance state info can be showed in webUI/JMX. But it
> can't be displayed via CLI. This JIRA will improve on this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11774) Ozone:KSM : add deleteVolume
[ https://issues.apache.org/jira/browse/HDFS-11774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030287#comment-16030287 ] Xiaoyu Yao commented on HDFS-11774: --- Thanks [~msingh] for working on this. The patch looks good to me overall. Just two minor issues. 1. MetadataManager#getIterator()/getVolumeRootKey() I would suggest we add Metadata#isVolumeEmpty() instead of raw DB interator for the MetadataManager interface. The DB implementation details can be abstracted from the interface by moving some of the implementation details using DBInterator from VolumeManagerImpl#deleteVolume (Line 296-301) into MetadataManagerImpl#isVolumeEmpty(). 2. ACL/permission check for volume deletion. We should check the owner and ACLs before allowing deleting volumes. I'm OK with just check the owner here or wait for the other ticket on checkVolumeAccess to fix this. > Ozone:KSM : add deleteVolume > > > Key: HDFS-11774 > URL: https://issues.apache.org/jira/browse/HDFS-11774 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh > Attachments: HDFS-11774-HDFS-7240.001.patch > > > Delete a volume if there are no buckets present inside the volume. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11383) String duplication in org.apache.hadoop.fs.BlockLocation
[
https://issues.apache.org/jira/browse/HDFS-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030277#comment-16030277
]
Misha Dmitriev commented on HDFS-11383:
---
I think since this code is not under active development and its data fields are
unlikely to change, it will be easier for now to just make a small change to
the existing code. HashCodeBuilder looks like a useful thing for a class with
many diverse data fields. But one still needs to remember all the data fields
to pass it as parameters (or use the alternative API that uses reflection
internally, which should be damn slow for one thing...) So I think right here
just fixing the existing hashCode() is a reasonable tradeoff, and I hope you
will agree with me.
> String duplication in org.apache.hadoop.fs.BlockLocation
>
>
> Key: HDFS-11383
> URL: https://issues.apache.org/jira/browse/HDFS-11383
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HDFS-11383.01.patch, HDFS-11383.02.patch, hs2-crash-2.txt
>
>
> I am working on Hive performance, investigating the problem of high memory
> pressure when (a) a table consists of a high number (thousands) of partitions
> and (b) multiple queries run against it concurrently. It turns out that a lot
> of memory is wasted due to data duplication. One source of duplicate strings
> is class org.apache.hadoop.fs.BlockLocation. Its fields such as storageIds,
> topologyPaths, hosts, names, may collectively use up to 6% of memory in my
> benchmark, causing (together with other problematic classes) a huge memory
> spike. Of these 6% of memory taken by BlockLocation strings, more than 5% are
> wasted due to duplication.
> I think we need to add calls to String.intern() in the BlockLocation
> constructor, like:
> {code}
> this.hosts = internStringsInArray(hosts);
> ...
> private void internStringsInArray(String[] sar) {
> for (int i = 0; i < sar.length; i++) {
> sar[i] = sar[i].intern();
> }
> }
> {code}
> String.intern() performs very well starting from JDK 7. I've found some
> articles explaining the progress that was made by the HotSpot JVM developers
> in this area, verified that with benchmarks myself, and finally added quite a
> bit of interning to one of the Cloudera products without any issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11903) Ozone: Cleaning up local storage when closing MiniOzoneCluster
[
https://issues.apache.org/jira/browse/HDFS-11903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030272#comment-16030272
]
Anu Engineer commented on HDFS-11903:
-
[~liuml07] It is pretty easy to make it support different locations, you can
create a new path ID(say a UUID) and append it to the current path. That way
you can create multiple MiniOzoneClusters if needed. We have not yet run into
the need to create multiple MiniOzoneClusters yet. Hence the current design.
> Ozone: Cleaning up local storage when closing MiniOzoneCluster
> --
>
> Key: HDFS-11903
> URL: https://issues.apache.org/jira/browse/HDFS-11903
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-11903-HDFS-7240.000.patch
>
>
> Currently we don't delete the local storage when closing
> {{MiniOzoneCluster}}, which will cause creating volume with the same name to
> fail, even in a new JVM process. Let's delete the local storage files:
> {{/tmp/ozone/metadata.db}} and {{/tmp/ozone/user.db}} when closing.
> Please note this is not a complete solution. As the {{OzoneMetadataManager}}
> currently is a singleton, we are not able to create two independent
> MiniOzoneCluster in a single JVM. Data maybe shared unexpectedly. Let's
> address that in another JIRA, if needed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11903) Ozone: Cleaning up local storage when closing MiniOzoneCluster
[
https://issues.apache.org/jira/browse/HDFS-11903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030269#comment-16030269
]
Anu Engineer commented on HDFS-11903:
-
+1, pending jenkins.
> Ozone: Cleaning up local storage when closing MiniOzoneCluster
> --
>
> Key: HDFS-11903
> URL: https://issues.apache.org/jira/browse/HDFS-11903
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-11903-HDFS-7240.000.patch
>
>
> Currently we don't delete the local storage when closing
> {{MiniOzoneCluster}}, which will cause creating volume with the same name to
> fail, even in a new JVM process. Let's delete the local storage files:
> {{/tmp/ozone/metadata.db}} and {{/tmp/ozone/user.db}} when closing.
> Please note this is not a complete solution. As the {{OzoneMetadataManager}}
> currently is a singleton, we are not able to create two independent
> MiniOzoneCluster in a single JVM. Data maybe shared unexpectedly. Let's
> address that in another JIRA, if needed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11903) Ozone: Cleaning up local storage when closing MiniOzoneCluster
[
https://issues.apache.org/jira/browse/HDFS-11903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-11903:
Summary: Ozone: Cleaning up local storage when closing MiniOzoneCluster
(was: Cleaning up local storage when closing MiniOzoneCluster)
> Ozone: Cleaning up local storage when closing MiniOzoneCluster
> --
>
> Key: HDFS-11903
> URL: https://issues.apache.org/jira/browse/HDFS-11903
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-11903-HDFS-7240.000.patch
>
>
> Currently we don't delete the local storage when closing
> {{MiniOzoneCluster}}, which will cause creating volume with the same name to
> fail, even in a new JVM process. Let's delete the local storage files:
> {{/tmp/ozone/metadata.db}} and {{/tmp/ozone/user.db}} when closing.
> Please note this is not a complete solution. As the {{OzoneMetadataManager}}
> currently is a singleton, we are not able to create two independent
> MiniOzoneCluster in a single JVM. Data maybe shared unexpectedly. Let's
> address that in another JIRA, if needed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11886) Ozone : improve error handling for putkey operation
[ https://issues.apache.org/jira/browse/HDFS-11886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11886: Attachment: design-notes-putkey.pdf [~xyao], [~szetszwo], [~cheersyang] uploading a doc that captures the off-line discussion that me and [~xyao] had. Please let me know if you have any questions. [~vagarychen] This is also useful when we have to support multi-stage keys -- That is keys which are uploaded as independent transactions but appear as a single key. Both s3 and azure supports that. > Ozone : improve error handling for putkey operation > --- > > Key: HDFS-11886 > URL: https://issues.apache.org/jira/browse/HDFS-11886 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Chen Liang > Attachments: design-notes-putkey.pdf > > > Ozone's putKey operations involve a couple steps: > 1. KSM calls allocateBlock to SCM, writes this info to KSM's local metastore > 2. allocatedBlock gets returned to client, client checks to see if container > needs to be created on datanode, if yes, create the container > 3. writes the data to container. > it is possible that 1 succeeded, but 2 or 3 failed, in this case there will > be an entry in KSM's local metastore, but the key is actually nowhere to be > found. We need to revert 1 is 2 or 3 failed in this case. > To resolve this, we need at least two things to be implemented first. > 1. We need deleteKey() to be added KSM first. > 2. We also need container reports to be implemented first such that SCM can > track whether the container is actually added. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11853) Ozone: KSM: Add getKey
[ https://issues.apache.org/jira/browse/HDFS-11853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11853: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Target Version/s: HDFS-7240 Status: Resolved (was: Patch Available) Thanks [~vagarychen] for the contribution. I've committed the patch to the feature branch. > Ozone: KSM: Add getKey > --- > > Key: HDFS-11853 > URL: https://issues.apache.org/jira/browse/HDFS-11853 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Chen Liang > Fix For: HDFS-7240 > > Attachments: HDFS-11853-HDFS-7240.001.patch, > HDFS-11853-HDFS-7240.002.patch, HDFS-11853-HDFS-7240.003.patch > > > Support read the content (object) of the key. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11853) Ozone: KSM: Add getKey
[ https://issues.apache.org/jira/browse/HDFS-11853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030243#comment-16030243 ] Xiaoyu Yao edited comment on HDFS-11853 at 5/30/17 10:15 PM: - Thanks [~vagarychen] for the update. Patch v3 looks good to me. +1, I will commit it shortly. was (Author: xyao): Thanks [~vagarychen] for the update. Patch v3 looks good to me. +1 > Ozone: KSM: Add getKey > --- > > Key: HDFS-11853 > URL: https://issues.apache.org/jira/browse/HDFS-11853 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Chen Liang > Attachments: HDFS-11853-HDFS-7240.001.patch, > HDFS-11853-HDFS-7240.002.patch, HDFS-11853-HDFS-7240.003.patch > > > Support read the content (object) of the key. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11853) Ozone: KSM: Add getKey
[ https://issues.apache.org/jira/browse/HDFS-11853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030243#comment-16030243 ] Xiaoyu Yao commented on HDFS-11853: --- Thanks [~vagarychen] for the update. Patch v3 looks good to me. +1 > Ozone: KSM: Add getKey > --- > > Key: HDFS-11853 > URL: https://issues.apache.org/jira/browse/HDFS-11853 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Chen Liang > Attachments: HDFS-11853-HDFS-7240.001.patch, > HDFS-11853-HDFS-7240.002.patch, HDFS-11853-HDFS-7240.003.patch > > > Support read the content (object) of the key. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11903) Cleaning up local storage when closing MiniOzoneCluster
[
https://issues.apache.org/jira/browse/HDFS-11903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-11903:
-
Status: Patch Available (was: Open)
> Cleaning up local storage when closing MiniOzoneCluster
> ---
>
> Key: HDFS-11903
> URL: https://issues.apache.org/jira/browse/HDFS-11903
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-11903-HDFS-7240.000.patch
>
>
> Currently we don't delete the local storage when closing
> {{MiniOzoneCluster}}, which will cause creating volume with the same name to
> fail, even in a new JVM process. Let's delete the local storage files:
> {{/tmp/ozone/metadata.db}} and {{/tmp/ozone/user.db}} when closing.
> Please note this is not a complete solution. As the {{OzoneMetadataManager}}
> currently is a singleton, we are not able to create two independent
> MiniOzoneCluster in a single JVM. Data maybe shared unexpectedly. Let's
> address that in another JIRA, if needed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11903) Cleaning up local storage when closing MiniOzoneCluster
[
https://issues.apache.org/jira/browse/HDFS-11903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030222#comment-16030222
]
Mingliang Liu commented on HDFS-11903:
--
Ping [~vagarychen] and [~xyao] for review. Thanks,
> Cleaning up local storage when closing MiniOzoneCluster
> ---
>
> Key: HDFS-11903
> URL: https://issues.apache.org/jira/browse/HDFS-11903
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-11903-HDFS-7240.000.patch
>
>
> Currently we don't delete the local storage when closing
> {{MiniOzoneCluster}}, which will cause creating volume with the same name to
> fail, even in a new JVM process. Let's delete the local storage files:
> {{/tmp/ozone/metadata.db}} and {{/tmp/ozone/user.db}} when closing.
> Please note this is not a complete solution. As the {{OzoneMetadataManager}}
> currently is a singleton, we are not able to create two independent
> MiniOzoneCluster in a single JVM. Data maybe shared unexpectedly. Let's
> address that in another JIRA, if needed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11903) Cleaning up local storage when closing MiniOzoneCluster
[
https://issues.apache.org/jira/browse/HDFS-11903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-11903:
-
Attachment: HDFS-11903-HDFS-7240.000.patch
> Cleaning up local storage when closing MiniOzoneCluster
> ---
>
> Key: HDFS-11903
> URL: https://issues.apache.org/jira/browse/HDFS-11903
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-11903-HDFS-7240.000.patch
>
>
> Currently we don't delete the local storage when closing
> {{MiniOzoneCluster}}, which will cause creating volume with the same name to
> fail, even in a new JVM process. Let's delete the local storage files:
> {{/tmp/ozone/metadata.db}} and {{/tmp/ozone/user.db}} when closing.
> Please note this is not a complete solution. As the {{OzoneMetadataManager}}
> currently is a singleton, we are not able to create two independent
> MiniOzoneCluster in a single JVM. Data maybe shared unexpectedly. Let's
> address that in another JIRA, if needed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11903) Cleaning up local storage when closing MiniOzoneCluster
Mingliang Liu created HDFS-11903:
Summary: Cleaning up local storage when closing MiniOzoneCluster
Key: HDFS-11903
URL: https://issues.apache.org/jira/browse/HDFS-11903
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Currently we don't delete the local storage when closing {{MiniOzoneCluster}},
which will cause creating volume with the same name to fail, even in a new JVM
process. Let's delete the local storage files: {{/tmp/ozone/metadata.db}} and
{{/tmp/ozone/user.db}} when closing.
Please note this is not a complete solution. As the {{OzoneMetadataManager}}
currently is a singleton, we are not able to create two independent
MiniOzoneCluster in a single JVM. Data maybe shared unexpectedly. Let's address
that in another JIRA, if needed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11853) Ozone: KSM: Add getKey
[ https://issues.apache.org/jira/browse/HDFS-11853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030183#comment-16030183 ] Chen Liang commented on HDFS-11853: --- Any further comments on v003 patch? shall we commit this? [~xyao]? [~anu]? > Ozone: KSM: Add getKey > --- > > Key: HDFS-11853 > URL: https://issues.apache.org/jira/browse/HDFS-11853 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Chen Liang > Attachments: HDFS-11853-HDFS-7240.001.patch, > HDFS-11853-HDFS-7240.002.patch, HDFS-11853-HDFS-7240.003.patch > > > Support read the content (object) of the key. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and TextFileRegionProvider into one.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Status: Patch Available (was: Open)
> [READ] Merge BlockFormatProvider and TextFileRegionProvider into one.
> -
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and TextFileRegionProvider into one.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Attachment: HDFS-11902-HDFS-9806.001.patch
Initial patch attached.
> [READ] Merge BlockFormatProvider and TextFileRegionProvider into one.
> -
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11902) [READ] Merge BlockFormatProvider and TextFileRegionProvider into one.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti reassigned HDFS-11902:
-
Assignee: Virajith Jalaparti
> [READ] Merge BlockFormatProvider and TextFileRegionProvider into one.
> -
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11383) String duplication in org.apache.hadoop.fs.BlockLocation
[
https://issues.apache.org/jira/browse/HDFS-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030164#comment-16030164
]
Andrew Wang commented on HDFS-11383:
Yea, maybe we should move to using an equals/hashCode builder? There's a nice
one in Apache Commons, Guava too. I'm generally against manually implemented
equals/hashCode since the builders exist.
> String duplication in org.apache.hadoop.fs.BlockLocation
>
>
> Key: HDFS-11383
> URL: https://issues.apache.org/jira/browse/HDFS-11383
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HDFS-11383.01.patch, HDFS-11383.02.patch, hs2-crash-2.txt
>
>
> I am working on Hive performance, investigating the problem of high memory
> pressure when (a) a table consists of a high number (thousands) of partitions
> and (b) multiple queries run against it concurrently. It turns out that a lot
> of memory is wasted due to data duplication. One source of duplicate strings
> is class org.apache.hadoop.fs.BlockLocation. Its fields such as storageIds,
> topologyPaths, hosts, names, may collectively use up to 6% of memory in my
> benchmark, causing (together with other problematic classes) a huge memory
> spike. Of these 6% of memory taken by BlockLocation strings, more than 5% are
> wasted due to duplication.
> I think we need to add calls to String.intern() in the BlockLocation
> constructor, like:
> {code}
> this.hosts = internStringsInArray(hosts);
> ...
> private void internStringsInArray(String[] sar) {
> for (int i = 0; i < sar.length; i++) {
> sar[i] = sar[i].intern();
> }
> }
> {code}
> String.intern() performs very well starting from JDK 7. I've found some
> articles explaining the progress that was made by the HotSpot JVM developers
> in this area, verified that with benchmarks myself, and finally added quite a
> bit of interning to one of the Cloudera products without any issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11902) [READ] Merge BlockFormatProvider and TextFileRegionProvider into one.
Virajith Jalaparti created HDFS-11902:
-
Summary: [READ] Merge BlockFormatProvider and
TextFileRegionProvider into one.
Key: HDFS-11902
URL: https://issues.apache.org/jira/browse/HDFS-11902
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Virajith Jalaparti
Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform almost
the same function on the Namenode and Datanode respectively. This JIRA is to
merge them into one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11383) String duplication in org.apache.hadoop.fs.BlockLocation
[
https://issues.apache.org/jira/browse/HDFS-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030151#comment-16030151
]
Misha Dmitriev commented on HDFS-11383:
---
Thank you Andrew. This findbugs warning makes sense, in principle, since there
is an ExtendedBlock constructor that indeed sets poolId to null. I've just
replaced all the simple 'poolId.intern()' statements with 'poolId != null ?
poolId.intern() : null'. But now I see that hashCode() and equals() would fail
if poolId is null. Do you think it makes sense to fix them just in case as well?
> String duplication in org.apache.hadoop.fs.BlockLocation
>
>
> Key: HDFS-11383
> URL: https://issues.apache.org/jira/browse/HDFS-11383
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HDFS-11383.01.patch, HDFS-11383.02.patch, hs2-crash-2.txt
>
>
> I am working on Hive performance, investigating the problem of high memory
> pressure when (a) a table consists of a high number (thousands) of partitions
> and (b) multiple queries run against it concurrently. It turns out that a lot
> of memory is wasted due to data duplication. One source of duplicate strings
> is class org.apache.hadoop.fs.BlockLocation. Its fields such as storageIds,
> topologyPaths, hosts, names, may collectively use up to 6% of memory in my
> benchmark, causing (together with other problematic classes) a huge memory
> spike. Of these 6% of memory taken by BlockLocation strings, more than 5% are
> wasted due to duplication.
> I think we need to add calls to String.intern() in the BlockLocation
> constructor, like:
> {code}
> this.hosts = internStringsInArray(hosts);
> ...
> private void internStringsInArray(String[] sar) {
> for (int i = 0; i < sar.length; i++) {
> sar[i] = sar[i].intern();
> }
> }
> {code}
> String.intern() performs very well starting from JDK 7. I've found some
> articles explaining the progress that was made by the HotSpot JVM developers
> in this area, verified that with benchmarks myself, and finally added quite a
> bit of interning to one of the Cloudera products without any issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11741) Long running balancer may fail due to expired DataEncryptionKey
[
https://issues.apache.org/jira/browse/HDFS-11741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030148#comment-16030148
]
Xiao Chen commented on HDFS-11741:
--
Looked again at this one, and I think {{encryptionKey.expiryDate <
timer.now()}} should be OK.
Found the graph isn't exactly accurate though, I think the timeline of the
block key goes like this:
generate --- (Tk) ---> becomes current --- (Tk (not Tl)) ---> retiring ---
(Tk + Tl) ---> expire (removed)
And the Encryption Key expires at Tl.
So with the buffer of (Tk) before block key removal, I think we're safe to only
compare Encryption Key's expiry as you said. :)
> Long running balancer may fail due to expired DataEncryptionKey
> ---
>
> Key: HDFS-11741
> URL: https://issues.apache.org/jira/browse/HDFS-11741
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
> Environment: CDH5.8.2, Kerberos, Data transfer encryption enabled.
> Balancer login using keytab
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: block keys.png, HDFS-11741.001.patch,
> HDFS-11741.002.patch, HDFS-11741.003.patch, HDFS-11741.004.patch,
> HDFS-11741.005.patch
>
>
> We found a long running balancer may fail despite using keytab, because
> KeyManager returns expired DataEncryptionKey, and it throws the following
> exception:
> {noformat}
> 2017-04-30 05:03:58,661 WARN [pool-1464-thread-10] balancer.Dispatcher
> (Dispatcher.java:dispatch(325)) - Failed to move blk_1067352712_3913241 with
> size=546650 from 10.0.0.134:50010:DISK to 10.0.0.98:50010:DISK through
> 10.0.0.134:50010
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=1005215027) doesn't exist. Current key: 1005215030
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:474)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:299)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:242)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:183)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.dispatch(Dispatcher.java:311)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$2300(Dispatcher.java:182)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$1.run(Dispatcher.java:899)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This bug is similar in nature to HDFS-10609. While balancer KeyManager
> actively synchronizes itself with NameNode w.r.t block keys, it does not
> update DataEncryptionKey accordingly.
> In a specific cluster, with Kerberos ticket life time 10 hours, and default
> block token expiration/life time 10 hours, a long running balancer failed
> after 20~30 hours.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11383) String duplication in org.apache.hadoop.fs.BlockLocation
[
https://issues.apache.org/jira/browse/HDFS-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-11383:
--
Status: In Progress (was: Patch Available)
> String duplication in org.apache.hadoop.fs.BlockLocation
>
>
> Key: HDFS-11383
> URL: https://issues.apache.org/jira/browse/HDFS-11383
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HDFS-11383.01.patch, HDFS-11383.02.patch, hs2-crash-2.txt
>
>
> I am working on Hive performance, investigating the problem of high memory
> pressure when (a) a table consists of a high number (thousands) of partitions
> and (b) multiple queries run against it concurrently. It turns out that a lot
> of memory is wasted due to data duplication. One source of duplicate strings
> is class org.apache.hadoop.fs.BlockLocation. Its fields such as storageIds,
> topologyPaths, hosts, names, may collectively use up to 6% of memory in my
> benchmark, causing (together with other problematic classes) a huge memory
> spike. Of these 6% of memory taken by BlockLocation strings, more than 5% are
> wasted due to duplication.
> I think we need to add calls to String.intern() in the BlockLocation
> constructor, like:
> {code}
> this.hosts = internStringsInArray(hosts);
> ...
> private void internStringsInArray(String[] sar) {
> for (int i = 0; i < sar.length; i++) {
> sar[i] = sar[i].intern();
> }
> }
> {code}
> String.intern() performs very well starting from JDK 7. I've found some
> articles explaining the progress that was made by the HotSpot JVM developers
> in this area, verified that with benchmarks myself, and finally added quite a
> bit of interning to one of the Cloudera products without any issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11774) Ozone:KSM : add deleteVolume
[
https://issues.apache.org/jira/browse/HDFS-11774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030083#comment-16030083
]
Hadoop QA commented on HDFS-11774:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
20s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
41s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
54s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
15s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 40s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
21s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}107m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11774 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12870414/HDFS-11774-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b2289423efa2 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 43febfa |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
[jira] [Commented] (HDFS-11383) String duplication in org.apache.hadoop.fs.BlockLocation
[
https://issues.apache.org/jira/browse/HDFS-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030075#comment-16030075
]
Andrew Wang commented on HDFS-11383:
Thanks for revving Misha, LGTM except we need to handle the new findbugs
warning in ExtendedBlock. I think adding a null check will fix it.
> String duplication in org.apache.hadoop.fs.BlockLocation
>
>
> Key: HDFS-11383
> URL: https://issues.apache.org/jira/browse/HDFS-11383
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HDFS-11383.01.patch, HDFS-11383.02.patch, hs2-crash-2.txt
>
>
> I am working on Hive performance, investigating the problem of high memory
> pressure when (a) a table consists of a high number (thousands) of partitions
> and (b) multiple queries run against it concurrently. It turns out that a lot
> of memory is wasted due to data duplication. One source of duplicate strings
> is class org.apache.hadoop.fs.BlockLocation. Its fields such as storageIds,
> topologyPaths, hosts, names, may collectively use up to 6% of memory in my
> benchmark, causing (together with other problematic classes) a huge memory
> spike. Of these 6% of memory taken by BlockLocation strings, more than 5% are
> wasted due to duplication.
> I think we need to add calls to String.intern() in the BlockLocation
> constructor, like:
> {code}
> this.hosts = internStringsInArray(hosts);
> ...
> private void internStringsInArray(String[] sar) {
> for (int i = 0; i < sar.length; i++) {
> sar[i] = sar[i].intern();
> }
> }
> {code}
> String.intern() performs very well starting from JDK 7. I've found some
> articles explaining the progress that was made by the HotSpot JVM developers
> in this area, verified that with benchmarks myself, and finally added quite a
> bit of interning to one of the Cloudera products without any issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11901) Modifier 'static' is redundant for inner enums less
[
https://issues.apache.org/jira/browse/HDFS-11901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030024#comment-16030024
]
Hadoop QA commented on HDFS-11901:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 45s{color} | {color:orange} hadoop-hdfs-project: The patch generated 7 new +
492 unchanged - 31 fixed = 499 total (was 523) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 25s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
30s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
52s{color} | {color:green} hadoop-hdfs-nfs in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
19s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11901 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12870367/HDFS-11901.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f27538d68000 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 62857be |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
|
[jira] [Commented] (HDFS-11853) Ozone: KSM: Add getKey
[ https://issues.apache.org/jira/browse/HDFS-11853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030007#comment-16030007 ] Chen Liang commented on HDFS-11853: --- The failed tests and the findbugs warnings are unrelated. > Ozone: KSM: Add getKey > --- > > Key: HDFS-11853 > URL: https://issues.apache.org/jira/browse/HDFS-11853 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Chen Liang > Attachments: HDFS-11853-HDFS-7240.001.patch, > HDFS-11853-HDFS-7240.002.patch, HDFS-11853-HDFS-7240.003.patch > > > Support read the content (object) of the key. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11659) TestDataNodeHotSwapVolumes.testRemoveVolumeBeingWritten fail due to no DataNode available for pipeline recovery.
[
https://issues.apache.org/jira/browse/HDFS-11659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029995#comment-16029995
]
Hudson commented on HDFS-11659:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11800 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11800/])
HDFS-11659. TestDataNodeHotSwapVolumes.testRemoveVolumeBeingWritten fail (lei:
rev 91d6fe151f2e3de21b0a9423ade921e771957d90)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
> TestDataNodeHotSwapVolumes.testRemoveVolumeBeingWritten fail due to no
> DataNode available for pipeline recovery.
>
>
> Key: HDFS-11659
> URL: https://issues.apache.org/jira/browse/HDFS-11659
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.3, 3.0.0-alpha2
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HDFS-11659.000.patch
>
>
> The test fails after the following error messages:
> {code}
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[127.0.0.1:57377,DS-b4ec61fc-657c-4e2a-9dc3-8d93b7769a2b,DISK],
>
> DatanodeInfoWithStorage[127.0.0.1:47448,DS-18bca8d7-048d-4d7f-9594-d2df16096a3d,DISK]],
>
> original=[DatanodeInfoWithStorage[127.0.0.1:57377,DS-b4ec61fc-657c-4e2a-9dc3-8d93b7769a2b,DISK],
>
> DatanodeInfoWithStorage[127.0.0.1:47448,DS-18bca8d7-048d-4d7f-9594-d2df16096a3d,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:1280)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1354)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1512)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1236)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:721)
> {code}
> In such case, the DataNode that has removed can not be used in the pipeline
> recovery.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10987) Make Decommission less expensive when lot of blocks present.
[ https://issues.apache.org/jira/browse/HDFS-10987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-10987: --- Fix Version/s: 2.7.4 > Make Decommission less expensive when lot of blocks present. > > > Key: HDFS-10987 > URL: https://issues.apache.org/jira/browse/HDFS-10987 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2 > > Attachments: HDFS-10987-002.patch, HDFS-10987-branch-2.7.patch, > HDFS-10987.patch > > > When user want to decommission a node which having 50M blocks +,it could hold > the namesystem lock for long time.We've seen it is taking 36 sec+. > As we knew during this time, Namenode will not available... As this > decommission will continuosly run till all the blocks got replicated,hence > Namenode will unavailable. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11883) [SPS] : Handle NPE in BlockStorageMovementTracker when dropSPSWork() called
[
https://issues.apache.org/jira/browse/HDFS-11883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029972#comment-16029972
]
Uma Maheswara Rao G commented on HDFS-11883:
{code}
if (blocksMoving.isEmpty() || moverTaskFutures.get(trackId) == null) {
{code}
This condition seems to work now. Because we allow only one time schedule per
file. One very corner case is, incase if this DN is not responded for long time
and NN started to reschedule and at the same time DN rejoined with blockReport.
Unfortunately NN selected the same DN again as co-ordinator, at this time DN
should dropSPSWork but current threads may continue. SO, while continuing that
old threads new future can be added. But in reality, this case may not happen
as future task may not be pending for that long. Even if it happens, old
results may be sent along with trackID. So, there should not be any leak I
think.
+1 on latest patch
> [SPS] : Handle NPE in BlockStorageMovementTracker when dropSPSWork() called
> ---
>
> Key: HDFS-11883
> URL: https://issues.apache.org/jira/browse/HDFS-11883
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-11883.001.patch, HDFS-11883-HDFS-10285.001.patch,
> HDFS-11883-HDFS-10285.002.patch
>
>
> {noformat}
> Exception in thread "BlockStorageMovementTracker"
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.BlockStorageMovementTracker.run(BlockStorageMovementTracker.java:91)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11771) Ozone: KSM: Add checkVolumeAccess
[
https://issues.apache.org/jira/browse/HDFS-11771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029958#comment-16029958
]
Anu Engineer commented on HDFS-11771:
-
[~msingh] Thanks for updating the patch. some very minor comments. Otherwise I
think we are good to go.
* KSMPbHelper.java:convertOzoneAcl
{code}
String userName;
if (acl.getName() == null) {
if (acl.getType() != OzoneAcl.OzoneACLType.WORLD) {
throw new IllegalArgumentException("Only root can have null username");
}
userName = "root";
} else {
userName = acl.getName();
}
{code}
I am not sure why we permit the null user at all ?
* nit: KsmVolumeArgs.java:ctor
Add new parameter to the arguments doc.
* OzoneMetadataManager.java:
The reason why we use ErrorTable.newError is to make sure that all web based
error message have a consitent format. If we need to add a newError that takes
a parameter, please feel free to add that to ErrorTable. This way REST API
error messages are consistent.
* VolumeArgs.java:
{{@param groups - list of groups user is part of}}
This comment seems wrong. It is the list of groups that is allowed to access
this volume.
> Ozone: KSM: Add checkVolumeAccess
> --
>
> Key: HDFS-11771
> URL: https://issues.apache.org/jira/browse/HDFS-11771
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-11771-HDFS-7240.001.patch,
> HDFS-11771-HDFS-7240.002.patch, HDFS-11771-HDFS-7240.003.patch
>
>
> Checks if the caller has access to a given volume. This call supports the
> ACLs specified in the ozone rest protocol documentation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11655) Ozone: CLI: Guarantees user runs SCM commands has appropriate permission
[ https://issues.apache.org/jira/browse/HDFS-11655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11655: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Status: Resolved (was: Patch Available) [~xyao] Thanks for the reviews. [~cheersyang] Thanks for the contribution. I have committed this to the feature branch. [~vagarychen] [~msingh] as noted earlier this can cause a cBlock deployment failure, if so we might have to make sure that cblock is running under the right user. > Ozone: CLI: Guarantees user runs SCM commands has appropriate permission > > > Key: HDFS-11655 > URL: https://issues.apache.org/jira/browse/HDFS-11655 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Labels: command-line, security > Fix For: HDFS-7240 > > Attachments: HDFS-11655-HDFS-7240.001.patch, > HDFS-11655-HDFS-7240.002.patch, HDFS-11655-HDFS-7240.003.patch, > HDFS-11655-HDFS-7240.004.patch > > > We need to add a permission check module for ozone command line utilities, to > make sure users run commands with proper privileges. For now, commands in > [design doc| > https://issues.apache.org/jira/secure/attachment/12861478/storage-container-manager-cli-v002.pdf] > all require admin privilege. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11659) TestDataNodeHotSwapVolumes.testRemoveVolumeBeingWritten fail due to no DataNode available for pipeline recovery.
[
https://issues.apache.org/jira/browse/HDFS-11659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-11659:
-
Resolution: Fixed
Fix Version/s: 3.0.0-alpha4
2.9.0
Status: Resolved (was: Patch Available)
Thanks for the reviews, [~jojochuang] and [~yzhangal].
> TestDataNodeHotSwapVolumes.testRemoveVolumeBeingWritten fail due to no
> DataNode available for pipeline recovery.
>
>
> Key: HDFS-11659
> URL: https://issues.apache.org/jira/browse/HDFS-11659
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.3, 3.0.0-alpha2
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HDFS-11659.000.patch
>
>
> The test fails after the following error messages:
> {code}
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[127.0.0.1:57377,DS-b4ec61fc-657c-4e2a-9dc3-8d93b7769a2b,DISK],
>
> DatanodeInfoWithStorage[127.0.0.1:47448,DS-18bca8d7-048d-4d7f-9594-d2df16096a3d,DISK]],
>
> original=[DatanodeInfoWithStorage[127.0.0.1:57377,DS-b4ec61fc-657c-4e2a-9dc3-8d93b7769a2b,DISK],
>
> DatanodeInfoWithStorage[127.0.0.1:47448,DS-18bca8d7-048d-4d7f-9594-d2df16096a3d,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:1280)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1354)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1512)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1236)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:721)
> {code}
> In such case, the DataNode that has removed can not be used in the pipeline
> recovery.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11887) XceiverClientManager should close XceiverClient on eviction from cache
[
https://issues.apache.org/jira/browse/HDFS-11887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029844#comment-16029844
]
Chen Liang commented on HDFS-11887:
---
Thanks [~msingh] for the patch! I have a question though. The original code has
the notion of reference count. Which is to handle the case where the cache
entry expires, but there is still active access to the connection. While seems
that in the patch, if it expires in the cache, it will be removed from the
cache and gets closed. So I wonder what happens if there is a long-lasting open
connection, it expires in the cache but it is used by someone. Will it be
problematic if we close the connection here?
> XceiverClientManager should close XceiverClient on eviction from cache
> --
>
> Key: HDFS-11887
> URL: https://issues.apache.org/jira/browse/HDFS-11887
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-11887-HDFS-7240.001.patch
>
>
> XceiverClientManager doesn't close client on eviction which can leak
> resources.
> {code}
> public XceiverClientManager(Configuration conf) {
> .
> .
> .
> public void onRemoval(
> RemovalNotification
> removalNotification) {
> // If the reference count is not 0, this xceiver client should
> not
> // be evicted, add it back to the cache.
> WithAccessInfo info = removalNotification.getValue();
> if (info.hasRefence()) {
> synchronized (XceiverClientManager.this.openClient) {
> XceiverClientManager.this
> .openClient.put(removalNotification.getKey(), info);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11774) Ozone:KSM : add deleteVolume
[ https://issues.apache.org/jira/browse/HDFS-11774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-11774: - Status: Patch Available (was: Open) > Ozone:KSM : add deleteVolume > > > Key: HDFS-11774 > URL: https://issues.apache.org/jira/browse/HDFS-11774 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh > Attachments: HDFS-11774-HDFS-7240.001.patch > > > Delete a volume if there are no buckets present inside the volume. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11774) Ozone:KSM : add deleteVolume
[ https://issues.apache.org/jira/browse/HDFS-11774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-11774: - Attachment: HDFS-11774-HDFS-7240.001.patch Initial patch for delete volume. > Ozone:KSM : add deleteVolume > > > Key: HDFS-11774 > URL: https://issues.apache.org/jira/browse/HDFS-11774 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh > Attachments: HDFS-11774-HDFS-7240.001.patch > > > Delete a volume if there are no buckets present inside the volume. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11901) Modifier 'static' is redundant for inner enums less
[
https://issues.apache.org/jira/browse/HDFS-11901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029808#comment-16029808
]
Brahma Reddy Battula commented on HDFS-11901:
-
[~linzhangbing] thanks for reporting this,added you HDFS contributor and
assigned to you.Now onwards you can assign the issues yourself.
Patch lgtm. {{test failures}} are unrelated..anyway triggered the jenkins again.
> Modifier 'static' is redundant for inner enums less
> ---
>
> Key: HDFS-11901
> URL: https://issues.apache.org/jira/browse/HDFS-11901
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha4
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
>Priority: Minor
> Attachments: HDFS-11901.001.patch
>
>
> Java enumeration type is a static constant, implicitly modified with static
> final,Modifier 'static' is redundant for inner enums less.So I suggest
> deleting the 'static' modifier.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5042) Completed files lost after power failure
[
https://issues.apache.org/jira/browse/HDFS-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029798#comment-16029798
]
Vinayakumar B commented on HDFS-5042:
-
For trunk patch, checkstyle can be ignored, as its inline with previous
indentation. Test failures are unrelated.
> Completed files lost after power failure
>
>
> Key: HDFS-5042
> URL: https://issues.apache.org/jira/browse/HDFS-5042
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: ext3 on CentOS 5.7 (kernel 2.6.18-274.el5)
>Reporter: Dave Latham
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-5042-01.patch, HDFS-5042-02.patch,
> HDFS-5042-03.patch, HDFS-5042-04.patch, HDFS-5042-05-branch-2.patch,
> HDFS-5042-05.patch, HDFS-5042-branch-2-01.patch, HDFS-5042-branch-2-05.patch
>
>
> We suffered a cluster wide power failure after which HDFS lost data that it
> had acknowledged as closed and complete.
> The client was HBase which compacted a set of HFiles into a new HFile, then
> after closing the file successfully, deleted the previous versions of the
> file. The cluster then lost power, and when brought back up the newly
> created file was marked CORRUPT.
> Based on reading the logs it looks like the replicas were created by the
> DataNodes in the 'blocksBeingWritten' directory. Then when the file was
> closed they were moved to the 'current' directory. After the power cycle
> those replicas were again in the blocksBeingWritten directory of the
> underlying file system (ext3). When those DataNodes reported in to the
> NameNode it deleted those replicas and lost the file.
> Some possible fixes could be having the DataNode fsync the directory(s) after
> moving the block from blocksBeingWritten to current to ensure the rename is
> durable or having the NameNode accept replicas from blocksBeingWritten under
> certain circumstances.
> Log snippets from RS (RegionServer), NN (NameNode), DN (DataNode):
> {noformat}
> RS 2013-06-29 11:16:06,812 DEBUG org.apache.hadoop.hbase.util.FSUtils:
> Creating
> file=hdfs://hm3:9000/hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> with permission=rwxrwxrwx
> NN 2013-06-29 11:16:06,830 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.allocateBlock:
> /hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c.
> blk_1395839728632046111_357084589
> DN 2013-06-29 11:16:06,832 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_1395839728632046111_357084589 src: /10.0.5.237:14327 dest:
> /10.0.5.237:50010
> NN 2013-06-29 11:16:11,370 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.0.6.1:50010 is added to
> blk_1395839728632046111_357084589 size 25418340
> NN 2013-06-29 11:16:11,370 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.0.6.24:50010 is added to
> blk_1395839728632046111_357084589 size 25418340
> NN 2013-06-29 11:16:11,385 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.0.5.237:50010 is added to
> blk_1395839728632046111_357084589 size 25418340
> DN 2013-06-29 11:16:11,385 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
> blk_1395839728632046111_357084589 of size 25418340 from /10.0.5.237:14327
> DN 2013-06-29 11:16:11,385 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 2 for block
> blk_1395839728632046111_357084589 terminating
> NN 2013-06-29 11:16:11,385 INFO org.apache.hadoop.hdfs.StateChange: Removing
> lease on file
> /hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> from client DFSClient_hb_rs_hs745,60020,1372470111932
> NN 2013-06-29 11:16:11,385 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.completeFile: file
> /hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> is closed by DFSClient_hb_rs_hs745,60020,1372470111932
> RS 2013-06-29 11:16:11,393 INFO org.apache.hadoop.hbase.regionserver.Store:
> Renaming compacted file at
> hdfs://hm3:9000/hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> to
> hdfs://hm3:9000/hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/n/6e0cc30af6e64e56ba5a539fdf159c4c
> RS 2013-06-29 11:16:11,505 INFO org.apache.hadoop.hbase.regionserver.Store:
> Completed major compaction of 7 file(s) in n of
> users-6,\x12\xBDp\xA3,1359426311784.b5b0820cde759ae68e333b2f4015bb7e. into
> 6e0cc30af6e64e56ba5a539fdf159c4c, size=24.2m; total size for store is 24.2m
> --- CRASH, RESTART -
> NN 2013-06-29 12:01:19,743 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
>
[jira] [Assigned] (HDFS-11901) Modifier 'static' is redundant for inner enums less
[ https://issues.apache.org/jira/browse/HDFS-11901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula reassigned HDFS-11901: --- Assignee: ZhangBing Lin > Modifier 'static' is redundant for inner enums less > --- > > Key: HDFS-11901 > URL: https://issues.apache.org/jira/browse/HDFS-11901 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin >Priority: Minor > Attachments: HDFS-11901.001.patch > > > Java enumeration type is a static constant, implicitly modified with static > final,Modifier 'static' is redundant for inner enums less.So I suggest > deleting the 'static' modifier. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5042) Completed files lost after power failure
[
https://issues.apache.org/jira/browse/HDFS-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029796#comment-16029796
]
Vinayakumar B commented on HDFS-5042:
-
bq. +1 on the syncing rbw on the fisrt hsync(). But let's focus on the
completed files in this jira and implement the extra safety in a separate one.
Let's get the branch-2 patch buttoned up.
Thanks [~kihwal].
Branch-2 patch is already attached, but jenkins is refusing to take it up.
Seems some problem in building docker image.
> Completed files lost after power failure
>
>
> Key: HDFS-5042
> URL: https://issues.apache.org/jira/browse/HDFS-5042
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: ext3 on CentOS 5.7 (kernel 2.6.18-274.el5)
>Reporter: Dave Latham
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-5042-01.patch, HDFS-5042-02.patch,
> HDFS-5042-03.patch, HDFS-5042-04.patch, HDFS-5042-05-branch-2.patch,
> HDFS-5042-05.patch, HDFS-5042-branch-2-01.patch, HDFS-5042-branch-2-05.patch
>
>
> We suffered a cluster wide power failure after which HDFS lost data that it
> had acknowledged as closed and complete.
> The client was HBase which compacted a set of HFiles into a new HFile, then
> after closing the file successfully, deleted the previous versions of the
> file. The cluster then lost power, and when brought back up the newly
> created file was marked CORRUPT.
> Based on reading the logs it looks like the replicas were created by the
> DataNodes in the 'blocksBeingWritten' directory. Then when the file was
> closed they were moved to the 'current' directory. After the power cycle
> those replicas were again in the blocksBeingWritten directory of the
> underlying file system (ext3). When those DataNodes reported in to the
> NameNode it deleted those replicas and lost the file.
> Some possible fixes could be having the DataNode fsync the directory(s) after
> moving the block from blocksBeingWritten to current to ensure the rename is
> durable or having the NameNode accept replicas from blocksBeingWritten under
> certain circumstances.
> Log snippets from RS (RegionServer), NN (NameNode), DN (DataNode):
> {noformat}
> RS 2013-06-29 11:16:06,812 DEBUG org.apache.hadoop.hbase.util.FSUtils:
> Creating
> file=hdfs://hm3:9000/hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> with permission=rwxrwxrwx
> NN 2013-06-29 11:16:06,830 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.allocateBlock:
> /hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c.
> blk_1395839728632046111_357084589
> DN 2013-06-29 11:16:06,832 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_1395839728632046111_357084589 src: /10.0.5.237:14327 dest:
> /10.0.5.237:50010
> NN 2013-06-29 11:16:11,370 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.0.6.1:50010 is added to
> blk_1395839728632046111_357084589 size 25418340
> NN 2013-06-29 11:16:11,370 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.0.6.24:50010 is added to
> blk_1395839728632046111_357084589 size 25418340
> NN 2013-06-29 11:16:11,385 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.0.5.237:50010 is added to
> blk_1395839728632046111_357084589 size 25418340
> DN 2013-06-29 11:16:11,385 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
> blk_1395839728632046111_357084589 of size 25418340 from /10.0.5.237:14327
> DN 2013-06-29 11:16:11,385 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 2 for block
> blk_1395839728632046111_357084589 terminating
> NN 2013-06-29 11:16:11,385 INFO org.apache.hadoop.hdfs.StateChange: Removing
> lease on file
> /hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> from client DFSClient_hb_rs_hs745,60020,1372470111932
> NN 2013-06-29 11:16:11,385 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.completeFile: file
> /hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> is closed by DFSClient_hb_rs_hs745,60020,1372470111932
> RS 2013-06-29 11:16:11,393 INFO org.apache.hadoop.hbase.regionserver.Store:
> Renaming compacted file at
> hdfs://hm3:9000/hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> to
> hdfs://hm3:9000/hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/n/6e0cc30af6e64e56ba5a539fdf159c4c
> RS 2013-06-29 11:16:11,505 INFO org.apache.hadoop.hbase.regionserver.Store:
> Completed major compaction of 7 file(s) in n of
> users-6,\x12\xBDp\xA3,1359426311784.b5b0820cde759ae68e333b2f4015bb7e. into
[jira] [Commented] (HDFS-5042) Completed files lost after power failure
[
https://issues.apache.org/jira/browse/HDFS-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029789#comment-16029789
]
Kihwal Lee commented on HDFS-5042:
--
+1 on the syncing rbw on the fisrt hsync(). But let's focus on the completed
files in this jira and implement the extra safety in a separate one. Let's get
the branch-2 patch buttoned up.
> Completed files lost after power failure
>
>
> Key: HDFS-5042
> URL: https://issues.apache.org/jira/browse/HDFS-5042
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: ext3 on CentOS 5.7 (kernel 2.6.18-274.el5)
>Reporter: Dave Latham
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-5042-01.patch, HDFS-5042-02.patch,
> HDFS-5042-03.patch, HDFS-5042-04.patch, HDFS-5042-05-branch-2.patch,
> HDFS-5042-05.patch, HDFS-5042-branch-2-01.patch, HDFS-5042-branch-2-05.patch
>
>
> We suffered a cluster wide power failure after which HDFS lost data that it
> had acknowledged as closed and complete.
> The client was HBase which compacted a set of HFiles into a new HFile, then
> after closing the file successfully, deleted the previous versions of the
> file. The cluster then lost power, and when brought back up the newly
> created file was marked CORRUPT.
> Based on reading the logs it looks like the replicas were created by the
> DataNodes in the 'blocksBeingWritten' directory. Then when the file was
> closed they were moved to the 'current' directory. After the power cycle
> those replicas were again in the blocksBeingWritten directory of the
> underlying file system (ext3). When those DataNodes reported in to the
> NameNode it deleted those replicas and lost the file.
> Some possible fixes could be having the DataNode fsync the directory(s) after
> moving the block from blocksBeingWritten to current to ensure the rename is
> durable or having the NameNode accept replicas from blocksBeingWritten under
> certain circumstances.
> Log snippets from RS (RegionServer), NN (NameNode), DN (DataNode):
> {noformat}
> RS 2013-06-29 11:16:06,812 DEBUG org.apache.hadoop.hbase.util.FSUtils:
> Creating
> file=hdfs://hm3:9000/hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> with permission=rwxrwxrwx
> NN 2013-06-29 11:16:06,830 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.allocateBlock:
> /hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c.
> blk_1395839728632046111_357084589
> DN 2013-06-29 11:16:06,832 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_1395839728632046111_357084589 src: /10.0.5.237:14327 dest:
> /10.0.5.237:50010
> NN 2013-06-29 11:16:11,370 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.0.6.1:50010 is added to
> blk_1395839728632046111_357084589 size 25418340
> NN 2013-06-29 11:16:11,370 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.0.6.24:50010 is added to
> blk_1395839728632046111_357084589 size 25418340
> NN 2013-06-29 11:16:11,385 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.0.5.237:50010 is added to
> blk_1395839728632046111_357084589 size 25418340
> DN 2013-06-29 11:16:11,385 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
> blk_1395839728632046111_357084589 of size 25418340 from /10.0.5.237:14327
> DN 2013-06-29 11:16:11,385 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 2 for block
> blk_1395839728632046111_357084589 terminating
> NN 2013-06-29 11:16:11,385 INFO org.apache.hadoop.hdfs.StateChange: Removing
> lease on file
> /hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> from client DFSClient_hb_rs_hs745,60020,1372470111932
> NN 2013-06-29 11:16:11,385 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.completeFile: file
> /hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> is closed by DFSClient_hb_rs_hs745,60020,1372470111932
> RS 2013-06-29 11:16:11,393 INFO org.apache.hadoop.hbase.regionserver.Store:
> Renaming compacted file at
> hdfs://hm3:9000/hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/.tmp/6e0cc30af6e64e56ba5a539fdf159c4c
> to
> hdfs://hm3:9000/hbase/users-6/b5b0820cde759ae68e333b2f4015bb7e/n/6e0cc30af6e64e56ba5a539fdf159c4c
> RS 2013-06-29 11:16:11,505 INFO org.apache.hadoop.hbase.regionserver.Store:
> Completed major compaction of 7 file(s) in n of
> users-6,\x12\xBDp\xA3,1359426311784.b5b0820cde759ae68e333b2f4015bb7e. into
> 6e0cc30af6e64e56ba5a539fdf159c4c, size=24.2m; total size for store is 24.2m
> --- CRASH, RESTART -
> NN 2013-06-29 12:01:19,743
[jira] [Comment Edited] (HDFS-11741) Long running balancer may fail due to expired DataEncryptionKey
[
https://issues.apache.org/jira/browse/HDFS-11741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026536#comment-16026536
]
Xiao Chen edited comment on HDFS-11741 at 5/30/17 5:23 PM:
---
Thanks for revving Wei-Chiu, good analysis!
As talked offline I think generating new DEK would be sufficient.
Prefer the {{encryptionKey.expiryDate - keyUpdateInterval / 4 * 3 <
timer.now()}} route to prevent TOCTOU as Andrew pointed out earlier.
Nits:
- {{LOG.debug("Getting new encryption token from NN");}} IIUC this is local
- Please remove the stale changes in {{TestBalancerWithEncryptedTransfer}} and
{{Dispatcher}}
- I think the test case in {{TestKeyManager}} needs updating after the 3/4
interval change - new DEK not generated based on expiry, but actually on BK's
update interval. Maybe we can choose different updateInterval and tokenLifetime
to differentiate it in the test.
- Let's use a safer test timeout to reduce false positives due to infra.
was (Author: xiaochen):
Thanks for revving Wei-Chiu, good analysis!
As talked offline I think generating new DEK would be sufficient.
Prefer the {{encryptionKey.expiryDate - keyUpdateInterval * 3 / 4 < timer.now()
}} route to prevent TOCTOU as Andrew pointed out earlier.
Nits:
- {{LOG.debug("Getting new encryption token from NN");}} IIUC this is local
- Please remove the space change in {{TestBalancerWithEncryptedTransfer}}
- I think the test case in {{TestKeyManager}} needs updating after the 3/4
interval change - new DEK not generated based on expiry, but actually on BK's
update interval. Maybe we can choose different updateInterval and tokenLifetime
to differentiate it in the test.
- Let's use a safer test timeout to reduce false positives due to infra.
> Long running balancer may fail due to expired DataEncryptionKey
> ---
>
> Key: HDFS-11741
> URL: https://issues.apache.org/jira/browse/HDFS-11741
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
> Environment: CDH5.8.2, Kerberos, Data transfer encryption enabled.
> Balancer login using keytab
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: block keys.png, HDFS-11741.001.patch,
> HDFS-11741.002.patch, HDFS-11741.003.patch, HDFS-11741.004.patch,
> HDFS-11741.005.patch
>
>
> We found a long running balancer may fail despite using keytab, because
> KeyManager returns expired DataEncryptionKey, and it throws the following
> exception:
> {noformat}
> 2017-04-30 05:03:58,661 WARN [pool-1464-thread-10] balancer.Dispatcher
> (Dispatcher.java:dispatch(325)) - Failed to move blk_1067352712_3913241 with
> size=546650 from 10.0.0.134:50010:DISK to 10.0.0.98:50010:DISK through
> 10.0.0.134:50010
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=1005215027) doesn't exist. Current key: 1005215030
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:474)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:299)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:242)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:183)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.dispatch(Dispatcher.java:311)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$2300(Dispatcher.java:182)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$1.run(Dispatcher.java:899)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This bug is similar in nature to HDFS-10609. While balancer KeyManager
> actively synchronizes itself with NameNode w.r.t block keys, it does not
> update DataEncryptionKey accordingly.
> In a specific cluster, with Kerberos ticket life time 10 hours, and default
> block token expiration/life time 10 hours, a long running balancer failed
> after 20~30 hours.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HDFS-11655) Ozone: CLI: Guarantees user runs SCM commands has appropriate permission
[ https://issues.apache.org/jira/browse/HDFS-11655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029746#comment-16029746 ] Anu Engineer commented on HDFS-11655: - +1, I am going to commit this shortly. [~vagarychen] [~msingh] This might cause failures in cblock deployment. if that happens, you might want to add cblock server to this group so it can talk to SCM. > Ozone: CLI: Guarantees user runs SCM commands has appropriate permission > > > Key: HDFS-11655 > URL: https://issues.apache.org/jira/browse/HDFS-11655 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Labels: command-line, security > Attachments: HDFS-11655-HDFS-7240.001.patch, > HDFS-11655-HDFS-7240.002.patch, HDFS-11655-HDFS-7240.003.patch, > HDFS-11655-HDFS-7240.004.patch > > > We need to add a permission check module for ozone command line utilities, to > make sure users run commands with proper privileges. For now, commands in > [design doc| > https://issues.apache.org/jira/secure/attachment/12861478/storage-container-manager-cli-v002.pdf] > all require admin privilege. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
