[jira] [Commented] (HDFS-12051) Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328411#comment-16328411
]
Yongjun Zhang commented on HDFS-12051:
--
HI [~szetszwo],
Thanks for your feedback. [~mi...@cloudera.com] changed the summary and
description to address your comments, so now they are consistent with the patch
changes. Wold you please take a look to see if it works for you? Thanks.
> Intern duplicate byte[] arrays (mainly those denoting file/directory names)
> to save memory
> --
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code:java}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries
[jira] [Commented] (HDFS-12990) Change default NameNode RPC port back to 8020
[ https://issues.apache.org/jira/browse/HDFS-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328380#comment-16328380 ] Tsz Wo Nicholas Sze commented on HDFS-12990: > ... the NN RPC port change is painful for downstream on migrating to Hadoop > 3. Some examples include ... Since this is a controversial issue, could we list out all the known problems caused by the port changes? Thanks. > Change default NameNode RPC port back to 8020 > - > > Key: HDFS-12990 > URL: https://issues.apache.org/jira/browse/HDFS-12990 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 3.0.0 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Critical > Attachments: HDFS-12990.01.patch > > > In HDFS-9427 (HDFS should not default to ephemeral ports), we changed all > default ports to ephemeral ports, which is very appreciated by admin. As part > of that change, we also modified the NN RPC port from the famous 8020 to > 9820, to be closer to other ports changed there. > With more integration going on, it appears that all the other ephemeral port > changes are fine, but the NN RPC port change is painful for downstream on > migrating to Hadoop 3. Some examples include: > # Hive table locations pointing to hdfs://nn:port/dir > # Downstream minicluster unit tests that assumed 8020 > # Oozie workflows / downstream scripts that used 8020 > This isn't a problem for HA URLs, since that does not include the port > number. But considering the downstream impact, instead of requiring all of > them change their stuff, it would be a way better experience to leave the NN > port unchanged. This will benefit Hadoop 3 adoption and ease unnecessary > upgrade burdens. > It is of course incompatible, but giving 3.0.0 is just out, IMO it worths to > switch the port back. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12522) Ozone: Remove the Priority Queues used in the Container State Manager
[
https://issues.apache.org/jira/browse/HDFS-12522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328368#comment-16328368
]
genericqa commented on HDFS-12522:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
5s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 49m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 69m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 43s{color}
| {color:red} hadoop-hdfs-project generated 12 new + 434 unchanged - 0 fixed =
446 total (was 434) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} hadoop-hdfs-project: The patch generated 11 new
+ 9 unchanged - 0 fixed = 20 total (was 9) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
38s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}123m 57s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
29s{color} | {color:red} The patch generated 6 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}299m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
| | hadoop.ozone.web.client.TestVolume |
| | hadoop.hdfs.TestFileAppend4 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.cli.TestHDFSCLI |
| | hadoop.ozone.container.common.TestDatanodeStateMachine |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| |
[jira] [Commented] (HDFS-12973) RBF: Document global quota supporting in federation
[
https://issues.apache.org/jira/browse/HDFS-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328360#comment-16328360
]
genericqa commented on HDFS-12973:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 20s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}120m 29s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
19s{color} | {color:red} The patch generated 23 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}164m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestHASafeMode |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.TestFileAppend4 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.server.namenode.TestAclConfigFlag |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.namenode.TestDeleteRace |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestDistributedFileSystemWithECFile |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | hadoop.hdfs.tools.TestStoragePolicyCommands |
| | hadoop.hdfs.TestParallelUnixDomainRead |
| | hadoop.hdfs.server.namenode.TestReencryption |
| | hadoop.hdfs.server.namenode.TestCheckpoint |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.namenode.snapshot.TestRandomOpsWithSnapshots |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| |
[jira] [Updated] (HDFS-12843) Ozone: Client: TestOzoneRpcClient#testPutKeyRatisThreeNodes is failing
[
https://issues.apache.org/jira/browse/HDFS-12843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-12843:
-
Status: Patch Available (was: Open)
> Ozone: Client: TestOzoneRpcClient#testPutKeyRatisThreeNodes is failing
> --
>
> Key: HDFS-12843
> URL: https://issues.apache.org/jira/browse/HDFS-12843
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nanda kumar
>Assignee: Mukul Kumar Singh
>Priority: Major
> Attachments: HDFS-12843-HDFS-7240.001.patch
>
>
> {{TestOzoneRpcClient#testPutKeyRatisThreeNodes}} is failing with below error
> {noformat}
> java.io.IOException: Create key failed, error:INTERNAL_ERROR
> at
> org.apache.hadoop.ozone.ksm.protocolPB.KeySpaceManagerProtocolClientSideTranslatorPB.openKey(KeySpaceManagerProtocolClientSideTranslatorPB.java:538)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:455)
> at
> org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:255)
> at
> org.apache.hadoop.ozone.client.rpc.TestOzoneRpcClient.testPutKeyRatisThreeNodes(TestOzoneRpcClient.java:487)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12843) Ozone: Client: TestOzoneRpcClient#testPutKeyRatisThreeNodes is failing
[
https://issues.apache.org/jira/browse/HDFS-12843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-12843:
-
Attachment: HDFS-12843-HDFS-7240.001.patch
> Ozone: Client: TestOzoneRpcClient#testPutKeyRatisThreeNodes is failing
> --
>
> Key: HDFS-12843
> URL: https://issues.apache.org/jira/browse/HDFS-12843
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nanda kumar
>Assignee: Mukul Kumar Singh
>Priority: Major
> Attachments: HDFS-12843-HDFS-7240.001.patch
>
>
> {{TestOzoneRpcClient#testPutKeyRatisThreeNodes}} is failing with below error
> {noformat}
> java.io.IOException: Create key failed, error:INTERNAL_ERROR
> at
> org.apache.hadoop.ozone.ksm.protocolPB.KeySpaceManagerProtocolClientSideTranslatorPB.openKey(KeySpaceManagerProtocolClientSideTranslatorPB.java:538)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:455)
> at
> org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:255)
> at
> org.apache.hadoop.ozone.client.rpc.TestOzoneRpcClient.testPutKeyRatisThreeNodes(TestOzoneRpcClient.java:487)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13012) Fix TestOzoneCOnfigurationFields
[ https://issues.apache.org/jira/browse/HDFS-13012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-13012: - Status: Open (was: Patch Available) > Fix TestOzoneCOnfigurationFields > > > Key: HDFS-13012 > URL: https://issues.apache.org/jira/browse/HDFS-13012 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Minor > Attachments: HDFS-13012-HDFS-7240.001.patch > > > "dfs.container.ratis.num.write.chunk.threads" and > "dfs.container.ratis.segment.size" were added with HDFS-12853, they need to > be added to ozone-default.xml to unblock test failures in > TestOzoneConfigurationFields. cc: [~msingh] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13012) Fix TestOzoneCOnfigurationFields
[ https://issues.apache.org/jira/browse/HDFS-13012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-13012: - Status: Patch Available (was: Open) > Fix TestOzoneCOnfigurationFields > > > Key: HDFS-13012 > URL: https://issues.apache.org/jira/browse/HDFS-13012 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Minor > Attachments: HDFS-13012-HDFS-7240.001.patch > > > "dfs.container.ratis.num.write.chunk.threads" and > "dfs.container.ratis.segment.size" were added with HDFS-12853, they need to > be added to ozone-default.xml to unblock test failures in > TestOzoneConfigurationFields. cc: [~msingh] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13012) Fix TestOzoneCOnfigurationFields
[ https://issues.apache.org/jira/browse/HDFS-13012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328343#comment-16328343 ] Mukul Kumar Singh commented on HDFS-13012: -- Thanks for working on this [~xyao]. +1, the patch looks good to me. Re-triggering jenkins as there are unrelated build failure in the last run. > Fix TestOzoneCOnfigurationFields > > > Key: HDFS-13012 > URL: https://issues.apache.org/jira/browse/HDFS-13012 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Minor > Attachments: HDFS-13012-HDFS-7240.001.patch > > > "dfs.container.ratis.num.write.chunk.threads" and > "dfs.container.ratis.segment.size" were added with HDFS-12853, they need to > be added to ozone-default.xml to unblock test failures in > TestOzoneConfigurationFields. cc: [~msingh] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up
[
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328345#comment-16328345
]
genericqa commented on HDFS-12935:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-12935 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-12935 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906347/HDFS-12935.007.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/22676/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Get ambiguous result for DFSAdmin command in HA mode when only one namenode
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.9.0, 3.0.0-beta1, 3.0.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Major
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch,
> HDFS-12935.004.patch, HDFS-12935.005.patch, HDFS-12935.006-branch.2.patch,
> HDFS-12935.006.patch, HDFS-12935.007-branch.2.patch, HDFS-12935.007.patch,
> HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When
> considering the following two occasions:
> (1)nn1 up and nn2 down
> (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin
> commands will have ambiguous results. The commands can be send successfully
> to the up namenode and are always functionally useful only when nn1 is up
> regardless of exception (IOException when connecting to the down namenode
> nn2). If only nn2 is up, the commands have no use at all and only exception
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to
> set balancer bandwidth value for datanodes as an example. It works and all
> the datanodes can get the setting values only when nn1 is up. If only nn2 is
> up, the command throws exception directly and no datanode get the bandwidth
> setting. Approximately ten DFSAdmin commands use the similar logical process
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to
> jiangjianfei02:9820 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to
> jiangjianfei01:9820 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]#
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13024) Ozone: ContainerStateMachine should synchronize operations between createContainer op and writeChunk
[
https://issues.apache.org/jira/browse/HDFS-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-13024:
-
Attachment: HDFS-13024-HDFS-7240.002.patch
> Ozone: ContainerStateMachine should synchronize operations between
> createContainer op and writeChunk
>
>
> Key: HDFS-13024
> URL: https://issues.apache.org/jira/browse/HDFS-13024
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13024-HDFS-7240.001.patch,
> HDFS-13024-HDFS-7240.002.patch
>
>
> This issue happens after HDFS-12853. with HDFS-12853, writeChunk op has been
> divided into two stages 1) the log write phase (here the state machine data
> is written) 2) ApplyTransaction.
> With a 3 node ratis ring, ratis leader will append the log entry to its log
> and forward it to its followers. However there is no guarantee on when the
> followers will apply the log to the state machine in {{applyTransaction}}.
> This issue happens in the following order
> 1) Leader accepts create container
> 2) Leader add entries to its logs and forwards to followers
> 3) Followers append the entry to its log and Ack to the raft leader (Please
> note that the transaction still hasn't been applied)
> 4) Leader applies the transaction and now replies
> 5) write chunk call is sent to the Leader
> 6) Leader now forwards the call to the followers
> 7) Followers try to apply the log by calling {{Dispatcher#dispatch}} however
> the create container call in 3) still hasn't been applied
> 8) write chunk call on followers fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up
[
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianfei Jiang updated HDFS-12935:
-
Status: Patch Available (was: In Progress)
Thanks [~brahmareddy] to point out. I have updated the patch to handle
\{{listOpenFiles}}. Please review if available.
> Get ambiguous result for DFSAdmin command in HA mode when only one namenode
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Affects Versions: 3.0.0, 3.0.0-beta1, 2.9.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Major
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch,
> HDFS-12935.004.patch, HDFS-12935.005.patch, HDFS-12935.006-branch.2.patch,
> HDFS-12935.006.patch, HDFS-12935.007-branch.2.patch, HDFS-12935.007.patch,
> HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When
> considering the following two occasions:
> (1)nn1 up and nn2 down
> (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin
> commands will have ambiguous results. The commands can be send successfully
> to the up namenode and are always functionally useful only when nn1 is up
> regardless of exception (IOException when connecting to the down namenode
> nn2). If only nn2 is up, the commands have no use at all and only exception
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to
> set balancer bandwidth value for datanodes as an example. It works and all
> the datanodes can get the setting values only when nn1 is up. If only nn2 is
> up, the command throws exception directly and no datanode get the bandwidth
> setting. Approximately ten DFSAdmin commands use the similar logical process
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to
> jiangjianfei02:9820 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to
> jiangjianfei01:9820 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]#
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up
[ https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-12935: - Attachment: HDFS-12935.007.patch > Get ambiguous result for DFSAdmin command in HA mode when only one namenode > is up > - > > Key: HDFS-12935 > URL: https://issues.apache.org/jira/browse/HDFS-12935 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.9.0, 3.0.0-beta1, 3.0.0 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, > HDFS-12935.004.patch, HDFS-12935.005.patch, HDFS-12935.006-branch.2.patch, > HDFS-12935.006.patch, HDFS-12935.007-branch.2.patch, HDFS-12935.007.patch, > HDFS_12935.001.patch > > > In HA mode, if one namenode is down, most of functions can still work. When > considering the following two occasions: > (1)nn1 up and nn2 down > (2)nn1 down and nn2 up > These two occasions should be equivalent. However, some of the DFSAdmin > commands will have ambiguous results. The commands can be send successfully > to the up namenode and are always functionally useful only when nn1 is up > regardless of exception (IOException when connecting to the down namenode > nn2). If only nn2 is up, the commands have no use at all and only exception > to connect nn1 can be found. > See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to > set balancer bandwidth value for datanodes as an example. It works and all > the datanodes can get the setting values only when nn1 is up. If only nn2 is > up, the command throws exception directly and no datanode get the bandwidth > setting. Approximately ten DFSAdmin commands use the similar logical process > and may be ambiguous. > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345 > *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820* > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei02:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234 > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei01:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up
[ https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-12935: - Attachment: HDFS-12935.007-branch.2.patch > Get ambiguous result for DFSAdmin command in HA mode when only one namenode > is up > - > > Key: HDFS-12935 > URL: https://issues.apache.org/jira/browse/HDFS-12935 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.9.0, 3.0.0-beta1, 3.0.0 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, > HDFS-12935.004.patch, HDFS-12935.005.patch, HDFS-12935.006-branch.2.patch, > HDFS-12935.006.patch, HDFS-12935.007-branch.2.patch, HDFS-12935.007.patch, > HDFS_12935.001.patch > > > In HA mode, if one namenode is down, most of functions can still work. When > considering the following two occasions: > (1)nn1 up and nn2 down > (2)nn1 down and nn2 up > These two occasions should be equivalent. However, some of the DFSAdmin > commands will have ambiguous results. The commands can be send successfully > to the up namenode and are always functionally useful only when nn1 is up > regardless of exception (IOException when connecting to the down namenode > nn2). If only nn2 is up, the commands have no use at all and only exception > to connect nn1 can be found. > See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to > set balancer bandwidth value for datanodes as an example. It works and all > the datanodes can get the setting values only when nn1 is up. If only nn2 is > up, the command throws exception directly and no datanode get the bandwidth > setting. Approximately ten DFSAdmin commands use the similar logical process > and may be ambiguous. > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345 > *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820* > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei02:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234 > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei01:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up
[ https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-12935: - Status: In Progress (was: Patch Available) > Get ambiguous result for DFSAdmin command in HA mode when only one namenode > is up > - > > Key: HDFS-12935 > URL: https://issues.apache.org/jira/browse/HDFS-12935 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 3.0.0, 3.0.0-beta1, 2.9.0 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, > HDFS-12935.004.patch, HDFS-12935.005.patch, HDFS-12935.006-branch.2.patch, > HDFS-12935.006.patch, HDFS-12935.007-branch.2.patch, HDFS-12935.007.patch, > HDFS_12935.001.patch > > > In HA mode, if one namenode is down, most of functions can still work. When > considering the following two occasions: > (1)nn1 up and nn2 down > (2)nn1 down and nn2 up > These two occasions should be equivalent. However, some of the DFSAdmin > commands will have ambiguous results. The commands can be send successfully > to the up namenode and are always functionally useful only when nn1 is up > regardless of exception (IOException when connecting to the down namenode > nn2). If only nn2 is up, the commands have no use at all and only exception > to connect nn1 can be found. > See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to > set balancer bandwidth value for datanodes as an example. It works and all > the datanodes can get the setting values only when nn1 is up. If only nn2 is > up, the command throws exception directly and no datanode get the bandwidth > setting. Approximately ten DFSAdmin commands use the similar logical process > and may be ambiguous. > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345 > *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820* > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei02:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234 > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei01:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12574) Add CryptoInputStream to WebHdfsFileSystem read call.
[
https://issues.apache.org/jira/browse/HDFS-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328308#comment-16328308
]
genericqa commented on HDFS-12574:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
49s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 34m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 72m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project: The patch generated 3 new +
354 unchanged - 2 fixed = 357 total (was 356) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
25s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 56s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}248m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestErasureCodingPolicies |
| | hadoop.hdfs.TestMaintenanceState |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.TestReplication |
| | hadoop.hdfs.server.federation.router.TestRouterRpc |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.hdfs.TestDFSStripedOutputStream |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| |
[jira] [Assigned] (HDFS-12843) Ozone: Client: TestOzoneRpcClient#testPutKeyRatisThreeNodes is failing
[
https://issues.apache.org/jira/browse/HDFS-12843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nanda kumar reassigned HDFS-12843:
--
Assignee: Mukul Kumar Singh (was: Nanda kumar)
> Ozone: Client: TestOzoneRpcClient#testPutKeyRatisThreeNodes is failing
> --
>
> Key: HDFS-12843
> URL: https://issues.apache.org/jira/browse/HDFS-12843
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nanda kumar
>Assignee: Mukul Kumar Singh
>Priority: Major
>
> {{TestOzoneRpcClient#testPutKeyRatisThreeNodes}} is failing with below error
> {noformat}
> java.io.IOException: Create key failed, error:INTERNAL_ERROR
> at
> org.apache.hadoop.ozone.ksm.protocolPB.KeySpaceManagerProtocolClientSideTranslatorPB.openKey(KeySpaceManagerProtocolClientSideTranslatorPB.java:538)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:455)
> at
> org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:255)
> at
> org.apache.hadoop.ozone.client.rpc.TestOzoneRpcClient.testPutKeyRatisThreeNodes(TestOzoneRpcClient.java:487)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13023) Journal Sync not working on a kerberos cluster
[
https://issues.apache.org/jira/browse/HDFS-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328305#comment-16328305
]
genericqa commented on HDFS-13023:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 30m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 21s{color}
| {color:red} root generated 1 new + 1239 unchanged - 0 fixed = 1240 total (was
1239) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 6s{color} | {color:orange} root: The patch generated 42 new + 104 unchanged
- 2 fixed = 146 total (was 106) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
7s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}111m 10s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
36s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}244m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13023 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-13024) Ozone: ContainerStateMachine should synchronize operations between createContainer op and writeChunk
[
https://issues.apache.org/jira/browse/HDFS-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328260#comment-16328260
]
genericqa commented on HDFS-13024:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
49s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 0 unchanged - 2 fixed = 2 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
21s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 18s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}161m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Synchronization performed on java.util.concurrent.ConcurrentHashMap in
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.triggerPostContainerCreate(String,
Message) At
ContainerStateMachine.java:org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.triggerPostContainerCreate(String,
Message) At ContainerStateMachine.java:[line 184] |
| | Synchronization performed on java.util.concurrent.ConcurrentHashMap in
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.writeStateMachineData(RaftProtos$LogEntryProto)
At
ContainerStateMachine.java:org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.writeStateMachineData(RaftProtos$LogEntryProto)
At ContainerStateMachine.java:[line 167] |
| Failed junit tests | hadoop.cblock.TestCBlockCLI |
| | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.ozone.scm.TestSCMCli |
| | hadoop.ozone.TestStorageContainerManager |
| |
[jira] [Comment Edited] (HDFS-12973) RBF: Document global quota supporting in federation
[ https://issues.apache.org/jira/browse/HDFS-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328246#comment-16328246 ] Yiqun Lin edited comment on HDFS-12973 at 1/17/18 4:06 AM: --- Thanks for the review, [~elgoiri]. Attach the updated patch for addressing Seems Jenkins didn't run well but I have generated pages in my local. It renders well and looks good. was (Author: linyiqun): Thanks for the review, [~elgoiri]. Attach the updated patch for addressing > RBF: Document global quota supporting in federation > --- > > Key: HDFS-12973 > URL: https://issues.apache.org/jira/browse/HDFS-12973 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Major > Attachments: HDFS-12973.001.patch, HDFS-12973.002.patch > > > Document global quota supporting in federation -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12973) RBF: Document global quota supporting in federation
[ https://issues.apache.org/jira/browse/HDFS-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328246#comment-16328246 ] Yiqun Lin commented on HDFS-12973: -- Thanks for the review, [~elgoiri]. Attach the updated patch for addressing > RBF: Document global quota supporting in federation > --- > > Key: HDFS-12973 > URL: https://issues.apache.org/jira/browse/HDFS-12973 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Major > Attachments: HDFS-12973.001.patch, HDFS-12973.002.patch > > > Document global quota supporting in federation -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12973) RBF: Document global quota supporting in federation
[ https://issues.apache.org/jira/browse/HDFS-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-12973: - Attachment: HDFS-12973.002.patch > RBF: Document global quota supporting in federation > --- > > Key: HDFS-12973 > URL: https://issues.apache.org/jira/browse/HDFS-12973 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Major > Attachments: HDFS-12973.001.patch, HDFS-12973.002.patch > > > Document global quota supporting in federation -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12990) Change default NameNode RPC port back to 8020
[ https://issues.apache.org/jira/browse/HDFS-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328191#comment-16328191 ] Eric Yang commented on HDFS-12990: -- [~chris.douglas] [~atm] This Jira makes no difference to me. If it saves people lives to revert a mistake, then by all means. I am not sure that we are saving people time and money to make rapid adjustment or create more work. I believe a vote would make people think about this problem and respond according to their preferences. This provides better data points for the community to respect outcome. That's all. > Change default NameNode RPC port back to 8020 > - > > Key: HDFS-12990 > URL: https://issues.apache.org/jira/browse/HDFS-12990 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 3.0.0 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Critical > Attachments: HDFS-12990.01.patch > > > In HDFS-9427 (HDFS should not default to ephemeral ports), we changed all > default ports to ephemeral ports, which is very appreciated by admin. As part > of that change, we also modified the NN RPC port from the famous 8020 to > 9820, to be closer to other ports changed there. > With more integration going on, it appears that all the other ephemeral port > changes are fine, but the NN RPC port change is painful for downstream on > migrating to Hadoop 3. Some examples include: > # Hive table locations pointing to hdfs://nn:port/dir > # Downstream minicluster unit tests that assumed 8020 > # Oozie workflows / downstream scripts that used 8020 > This isn't a problem for HA URLs, since that does not include the port > number. But considering the downstream impact, instead of requiring all of > them change their stuff, it would be a way better experience to leave the NN > port unchanged. This will benefit Hadoop 3 adoption and ease unnecessary > upgrade burdens. > It is of course incompatible, but giving 3.0.0 is just out, IMO it worths to > switch the port back. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12051) Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-12051:
--
Description:
When snapshot diff operation is performed in a NameNode that manages several
million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
result in 6.5% memory overhead, and most of these arrays are referenced by
{{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
{code:java}
19. DUPLICATE PRIMITIVE ARRAYS
Types of duplicate objects:
Ovhd Num objs Num unique objs Class name
3,220,272K (6.5%) 104749528 25760871 byte[]
1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487 of
byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114, 116,
45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51, 95,
48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
... and 45902395 more arrays, of which 13158084 are unique
<--
org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
<-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode <--
{j.u.ArrayList} <--
org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
<-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1 elements)
... <-- org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
<-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
409,830K (0.8%), 13482787 dup arrays (13260241 unique)
430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of byte[32](116,
97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of byte[32](116, 97, 115, 107,
95, 49, 52, 57, 55, 48, ...), 342 of byte[32](116, 97, 115, 107, 95, 49, 52,
57, 55, 48, ...), 341 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48,
...), 341 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of byte[32](116,
97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of byte[32](116, 97, 115, 107,
95, 49, 52, 57, 55, 48, ...)
... and 13479257 more arrays, of which 13260231 are unique
<-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
<-- j.l.Thread[] <--
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
<-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
{code}
There are several other places in NameNode code which also produce duplicate
{{byte[]}} arrays.
To eliminate this duplication and reclaim memory, we will need to write a small
class similar to StringInterner, but designed specifically for byte[] arrays.
was:
When snapshot diff operation is performed in a NameNode that manages several
million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
result in 6.5% memory overhead, and most of these arrays are referenced by
[jira] [Commented] (HDFS-12051) Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328189#comment-16328189
]
Misha Dmitriev commented on HDFS-12051:
---
I've just modified the ticket name and description.
> Intern duplicate byte[] arrays (mainly those denoting file/directory names)
> to save memory
> --
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code:java}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
>
[jira] [Updated] (HDFS-12051) Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-12051:
--
Summary: Intern duplicate byte[] arrays (mainly those denoting
file/directory names) to save memory (was: Intern duplicate byte[] arrays,
mainly denoting file/directory names, to save memory)
> Intern duplicate byte[] arrays (mainly those denoting file/directory names)
> to save memory
> --
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
>
[jira] [Updated] (HDFS-12051) Intern duplicate byte[] arrays, mainly denoting file/directory names, to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-12051:
--
Summary: Intern duplicate byte[] arrays, mainly denoting file/directory
names, to save memory (was: Intern INOdeFileAttributes$SnapshotCopy.name
byte[] arrays to save memory)
> Intern duplicate byte[] arrays, mainly denoting file/directory names, to save
> memory
>
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
>
[jira] [Commented] (HDFS-13013) Fix closeContainer API with the right container state change
[
https://issues.apache.org/jira/browse/HDFS-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328158#comment-16328158
]
genericqa commented on HDFS-13013:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
54s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
17s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
19s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 2
unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
57s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}139m 44s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}232m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeysRatis |
| | hadoop.ozone.ozShell.TestOzoneShell |
| | hadoop.ozone.TestOzoneConfigurationFields |
| | hadoop.hdfs.TestDFSRollback |
| | hadoop.ozone.container.common.impl.TestContainerPersistence |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.ozone.client.rpc.TestOzoneRpcClient |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestReadStripedFileWithDecodingCorruptData |
| |
[jira] [Commented] (HDFS-12051) Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328130#comment-16328130
]
Misha Dmitriev commented on HDFS-12051:
---
[~szetszwo] did you see my previous response, by any chance? If you want to
change this description, could you please be so kind to suggest your variant?
Otherwise, as I am really afraid based on the current experience interacting
with you, we may spend a lot more time just me suggesting new names and you
rejecting them.
> Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
>
[jira] [Commented] (HDFS-12051) Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328100#comment-16328100
]
Tsz Wo Nicholas Sze commented on HDFS-12051:
No. I still have at least the following question unanswered.
{quote}
Do you think that it is appropriate to claim that this is a SnapshotCopy change
in the summary and the description but the patch actually changes NameNode
internal silently, [~yzhangal]?
{quote}
> Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
>
[jira] [Commented] (HDFS-12990) Change default NameNode RPC port back to 8020
[ https://issues.apache.org/jira/browse/HDFS-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328096#comment-16328096 ] Tsz Wo Nicholas Sze commented on HDFS-12990: How about supporting both ports at the same time by default? It is compatible to both 2.x and 3.0.0. Of course, users could set it back to one of the ports in the conf. > Change default NameNode RPC port back to 8020 > - > > Key: HDFS-12990 > URL: https://issues.apache.org/jira/browse/HDFS-12990 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 3.0.0 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Critical > Attachments: HDFS-12990.01.patch > > > In HDFS-9427 (HDFS should not default to ephemeral ports), we changed all > default ports to ephemeral ports, which is very appreciated by admin. As part > of that change, we also modified the NN RPC port from the famous 8020 to > 9820, to be closer to other ports changed there. > With more integration going on, it appears that all the other ephemeral port > changes are fine, but the NN RPC port change is painful for downstream on > migrating to Hadoop 3. Some examples include: > # Hive table locations pointing to hdfs://nn:port/dir > # Downstream minicluster unit tests that assumed 8020 > # Oozie workflows / downstream scripts that used 8020 > This isn't a problem for HA URLs, since that does not include the port > number. But considering the downstream impact, instead of requiring all of > them change their stuff, it would be a way better experience to leave the NN > port unchanged. This will benefit Hadoop 3 adoption and ease unnecessary > upgrade burdens. > It is of course incompatible, but giving 3.0.0 is just out, IMO it worths to > switch the port back. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12990) Change default NameNode RPC port back to 8020
[
https://issues.apache.org/jira/browse/HDFS-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328056#comment-16328056
]
Aaron T. Myers commented on HDFS-12990:
---
{quote}I also don't like the port number, but I don't like the risk that
someone might be testing Hadoop 3.0.0 release, and decided to put 3.0.1 on
production at random future time to find that that we made an incompatible
change for NN rpc in a future event that we can not predict.
{quote}
This is a very pragmatic concern, but I'm hoping you'll agree that there are
*vastly* more users on 2.x versions that will eventually upgrade to some future
3.x version. Those 2.x users are also surely not expecting the default NN RPC
port to change for no good reason at all. To not fix this issue now will be
creating a headache for all of us, and likely close to 100% of our users
eventually, for years to come. Changing this back now is certainly not ideal,
but from a purely pragmatic perspective it is obviously the right choice.
> Change default NameNode RPC port back to 8020
> -
>
> Key: HDFS-12990
> URL: https://issues.apache.org/jira/browse/HDFS-12990
> Project: Hadoop HDFS
> Issue Type: Task
> Components: namenode
>Affects Versions: 3.0.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-12990.01.patch
>
>
> In HDFS-9427 (HDFS should not default to ephemeral ports), we changed all
> default ports to ephemeral ports, which is very appreciated by admin. As part
> of that change, we also modified the NN RPC port from the famous 8020 to
> 9820, to be closer to other ports changed there.
> With more integration going on, it appears that all the other ephemeral port
> changes are fine, but the NN RPC port change is painful for downstream on
> migrating to Hadoop 3. Some examples include:
> # Hive table locations pointing to hdfs://nn:port/dir
> # Downstream minicluster unit tests that assumed 8020
> # Oozie workflows / downstream scripts that used 8020
> This isn't a problem for HA URLs, since that does not include the port
> number. But considering the downstream impact, instead of requiring all of
> them change their stuff, it would be a way better experience to leave the NN
> port unchanged. This will benefit Hadoop 3 adoption and ease unnecessary
> upgrade burdens.
> It is of course incompatible, but giving 3.0.0 is just out, IMO it worths to
> switch the port back.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13025) [SPS]: Implement a mechanism to scan the files for external SPS
Uma Maheswara Rao G created HDFS-13025: -- Summary: [SPS]: Implement a mechanism to scan the files for external SPS Key: HDFS-13025 URL: https://issues.apache.org/jira/browse/HDFS-13025 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G HDFS-12911 modularization is introducing FileIDCollector interface for canning the files. That will help us to plugin different ways of scanning mechanisms if needed. For Internal SPS, we have INode based scanning. For external SPS, we should go via client API scanning as we can not access NN internal structures. This is the task to implement the scanning plugin for external SPS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12990) Change default NameNode RPC port back to 8020
[ https://issues.apache.org/jira/browse/HDFS-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328025#comment-16328025 ] Chris Douglas commented on HDFS-12990: -- [~eyang], that paragraph begs the question. Paraphrasing: "we must retain the change, because we can only work around the consequences." In every example you cite, the project and users got some benefit. Reverting this change _is_ an admissible solution to fix the bugs it caused. Our releases solve problems for our users. We're not here to curate a version control system and a set of compatibility rules. > Change default NameNode RPC port back to 8020 > - > > Key: HDFS-12990 > URL: https://issues.apache.org/jira/browse/HDFS-12990 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 3.0.0 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Critical > Attachments: HDFS-12990.01.patch > > > In HDFS-9427 (HDFS should not default to ephemeral ports), we changed all > default ports to ephemeral ports, which is very appreciated by admin. As part > of that change, we also modified the NN RPC port from the famous 8020 to > 9820, to be closer to other ports changed there. > With more integration going on, it appears that all the other ephemeral port > changes are fine, but the NN RPC port change is painful for downstream on > migrating to Hadoop 3. Some examples include: > # Hive table locations pointing to hdfs://nn:port/dir > # Downstream minicluster unit tests that assumed 8020 > # Oozie workflows / downstream scripts that used 8020 > This isn't a problem for HA URLs, since that does not include the port > number. But considering the downstream impact, instead of requiring all of > them change their stuff, it would be a way better experience to leave the NN > port unchanged. This will benefit Hadoop 3 adoption and ease unnecessary > upgrade burdens. > It is of course incompatible, but giving 3.0.0 is just out, IMO it worths to > switch the port back. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12574) Add CryptoInputStream to WebHdfsFileSystem read call.
[ https://issues.apache.org/jira/browse/HDFS-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-12574: -- Status: Patch Available (was: Open) > Add CryptoInputStream to WebHdfsFileSystem read call. > - > > Key: HDFS-12574 > URL: https://issues.apache.org/jira/browse/HDFS-12574 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: encryption, kms, webhdfs >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Major > Attachments: HDFS-12574.001.patch, HDFS-12574.002.patch, > HDFS-12574.003.patch, HDFS-12574.004.patch, HDFS-12574.005.patch, > HDFS-12574.006.patch, HDFS-12574.007.patch, HDFS-12574.008.patch, > HDFS-12574.009.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12574) Add CryptoInputStream to WebHdfsFileSystem read call.
[ https://issues.apache.org/jira/browse/HDFS-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-12574: -- Status: Open (was: Patch Available) > Add CryptoInputStream to WebHdfsFileSystem read call. > - > > Key: HDFS-12574 > URL: https://issues.apache.org/jira/browse/HDFS-12574 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: encryption, kms, webhdfs >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Major > Attachments: HDFS-12574.001.patch, HDFS-12574.002.patch, > HDFS-12574.003.patch, HDFS-12574.004.patch, HDFS-12574.005.patch, > HDFS-12574.006.patch, HDFS-12574.007.patch, HDFS-12574.008.patch, > HDFS-12574.009.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13004) TestLeaseRecoveryStriped#testLeaseRecovery is failing when safeLength is 0MB or larger than the test file
[
https://issues.apache.org/jira/browse/HDFS-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328020#comment-16328020
]
Hudson commented on HDFS-13004:
---
ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #13505 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13505/])
HDFS-13004. TestLeaseRecoveryStriped.testLeaseRecovery is failing when (lei:
rev 3bd9ea63df769345a9d02a404cfb61323a4cd7e3)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRecoveryStriped.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/StripedFileTestUtil.java
> TestLeaseRecoveryStriped#testLeaseRecovery is failing when safeLength is 0MB
> or larger than the test file
> -
>
> Key: HDFS-13004
> URL: https://issues.apache.org/jira/browse/HDFS-13004
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Zsolt Venczel
>Assignee: Zsolt Venczel
>Priority: Major
> Labels: flaky-test
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HDFS-13004.01.patch, HDFS-13004.02.patch,
> HDFS-13004.03.patch
>
>
> andre{code}
> Error:
> failed testCase at i=1,
> blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths=
> {4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304},safeLength=25165824]
> java.lang.AssertionError: File length should be the same expected:<25165824>
> but was:<18874368>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLength(StripedFileTestUtil.java:79)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.checkData(StripedFileTestUtil.java:362)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:198)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.testLeaseRecovery(TestLeaseRecoveryStriped.java:182)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> Stack:
> java.lang.AssertionError:
> failed testCase at i=1,
> blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths={4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304}
> ,safeLength=25165824]
> java.lang.AssertionError: File length should be the same expected:<25165824>
> but was:<18874368>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
>
[jira] [Updated] (HDFS-13004) TestLeaseRecoveryStriped#testLeaseRecovery is failing when safeLength is 0MB or larger than the test file
[
https://issues.apache.org/jira/browse/HDFS-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-13004:
-
Resolution: Fixed
Fix Version/s: 3.1.0
Target Version/s: 3.1.0, 3.0.1 (was: 3.0.0)
Status: Resolved (was: Patch Available)
Thanks [~zvenczel] for the contribution.
Committed to trunk and branch 3
> TestLeaseRecoveryStriped#testLeaseRecovery is failing when safeLength is 0MB
> or larger than the test file
> -
>
> Key: HDFS-13004
> URL: https://issues.apache.org/jira/browse/HDFS-13004
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Zsolt Venczel
>Assignee: Zsolt Venczel
>Priority: Major
> Labels: flaky-test
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HDFS-13004.01.patch, HDFS-13004.02.patch,
> HDFS-13004.03.patch
>
>
> andre{code}
> Error:
> failed testCase at i=1,
> blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths=
> {4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304},safeLength=25165824]
> java.lang.AssertionError: File length should be the same expected:<25165824>
> but was:<18874368>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLength(StripedFileTestUtil.java:79)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.checkData(StripedFileTestUtil.java:362)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:198)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.testLeaseRecovery(TestLeaseRecoveryStriped.java:182)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> Stack:
> java.lang.AssertionError:
> failed testCase at i=1,
> blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths={4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304}
> ,safeLength=25165824]
> java.lang.AssertionError: File length should be the same expected:<25165824>
> but was:<18874368>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLength(StripedFileTestUtil.java:79)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.checkData(StripedFileTestUtil.java:362)
> at
>
[jira] [Updated] (HDFS-12574) Add CryptoInputStream to WebHdfsFileSystem read call.
[ https://issues.apache.org/jira/browse/HDFS-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-12574: -- Attachment: HDFS-12574.009.patch > Add CryptoInputStream to WebHdfsFileSystem read call. > - > > Key: HDFS-12574 > URL: https://issues.apache.org/jira/browse/HDFS-12574 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: encryption, kms, webhdfs >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Major > Attachments: HDFS-12574.001.patch, HDFS-12574.002.patch, > HDFS-12574.003.patch, HDFS-12574.004.patch, HDFS-12574.005.patch, > HDFS-12574.006.patch, HDFS-12574.007.patch, HDFS-12574.008.patch, > HDFS-12574.009.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12522) Ozone: Remove the Priority Queues used in the Container State Manager
[ https://issues.apache.org/jira/browse/HDFS-12522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12522: Status: Patch Available (was: Open) [~elek] ,[~xyao], [~nandakumar131] Please take a look when you get a chance. > Ozone: Remove the Priority Queues used in the Container State Manager > - > > Key: HDFS-12522 > URL: https://issues.apache.org/jira/browse/HDFS-12522 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Major > Attachments: HDFS-12522-HDFS-7240.001.patch > > > During code review of HDFS-12387, it was suggested that we remove the > priority queues that was used in ContainerStateManager. This JIRA tracks that > issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12522) Ozone: Remove the Priority Queues used in the Container State Manager
[ https://issues.apache.org/jira/browse/HDFS-12522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12522: Attachment: HDFS-12522-HDFS-7240.001.patch > Ozone: Remove the Priority Queues used in the Container State Manager > - > > Key: HDFS-12522 > URL: https://issues.apache.org/jira/browse/HDFS-12522 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Major > Attachments: HDFS-12522-HDFS-7240.001.patch > > > During code review of HDFS-12387, it was suggested that we remove the > priority queues that was used in ContainerStateManager. This JIRA tracks that > issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13024) Ozone: ContainerStateMachine should synchronize operations between createContainer op and writeChunk
[
https://issues.apache.org/jira/browse/HDFS-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-13024:
-
Status: Patch Available (was: Open)
> Ozone: ContainerStateMachine should synchronize operations between
> createContainer op and writeChunk
>
>
> Key: HDFS-13024
> URL: https://issues.apache.org/jira/browse/HDFS-13024
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13024-HDFS-7240.001.patch
>
>
> This issue happens after HDFS-12853. with HDFS-12853, writeChunk op has been
> divided into two stages 1) the log write phase (here the state machine data
> is written) 2) ApplyTransaction.
> With a 3 node ratis ring, ratis leader will append the log entry to its log
> and forward it to its followers. However there is no guarantee on when the
> followers will apply the log to the state machine in {{applyTransaction}}.
> This issue happens in the following order
> 1) Leader accepts create container
> 2) Leader add entries to its logs and forwards to followers
> 3) Followers append the entry to its log and Ack to the raft leader (Please
> note that the transaction still hasn't been applied)
> 4) Leader applies the transaction and now replies
> 5) write chunk call is sent to the Leader
> 6) Leader now forwards the call to the followers
> 7) Followers try to apply the log by calling {{Dispatcher#dispatch}} however
> the create container call in 3) still hasn't been applied
> 8) write chunk call on followers fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13024) Ozone: ContainerStateMachine should synchronize operations between createContainer op and writeChunk
[
https://issues.apache.org/jira/browse/HDFS-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-13024:
-
Attachment: HDFS-13024-HDFS-7240.001.patch
> Ozone: ContainerStateMachine should synchronize operations between
> createContainer op and writeChunk
>
>
> Key: HDFS-13024
> URL: https://issues.apache.org/jira/browse/HDFS-13024
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13024-HDFS-7240.001.patch
>
>
> This issue happens after HDFS-12853. with HDFS-12853, writeChunk op has been
> divided into two stages 1) the log write phase (here the state machine data
> is written) 2) ApplyTransaction.
> With a 3 node ratis ring, ratis leader will append the log entry to its log
> and forward it to its followers. However there is no guarantee on when the
> followers will apply the log to the state machine in {{applyTransaction}}.
> This issue happens in the following order
> 1) Leader accepts create container
> 2) Leader add entries to its logs and forwards to followers
> 3) Followers append the entry to its log and Ack to the raft leader (Please
> note that the transaction still hasn't been applied)
> 4) Leader applies the transaction and now replies
> 5) write chunk call is sent to the Leader
> 6) Leader now forwards the call to the followers
> 7) Followers try to apply the log by calling {{Dispatcher#dispatch}} however
> the create container call in 3) still hasn't been applied
> 8) write chunk call on followers fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13024) Ozone: ContainerStateMachine should synchronize operations between createContainer op and writeChunk
Mukul Kumar Singh created HDFS-13024:
Summary: Ozone: ContainerStateMachine should synchronize
operations between createContainer op and writeChunk
Key: HDFS-13024
URL: https://issues.apache.org/jira/browse/HDFS-13024
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Affects Versions: HDFS-7240
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh
Fix For: HDFS-7240
This issue happens after HDFS-12853. with HDFS-12853, writeChunk op has been
divided into two stages 1) the log write phase (here the state machine data is
written) 2) ApplyTransaction.
With a 3 node ratis ring, ratis leader will append the log entry to its log and
forward it to its followers. However there is no guarantee on when the
followers will apply the log to the state machine in {{applyTransaction}}.
This issue happens in the following order
1) Leader accepts create container
2) Leader add entries to its logs and forwards to followers
3) Followers append the entry to its log and Ack to the raft leader (Please
note that the transaction still hasn't been applied)
4) Leader applies the transaction and now replies
5) write chunk call is sent to the Leader
6) Leader now forwards the call to the followers
7) Followers try to apply the log by calling {{Dispatcher#dispatch}} however
the create container call in 3) still hasn't been applied
8) write chunk call on followers fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13023) Journal Sync not working on a kerberos cluster
[
https://issues.apache.org/jira/browse/HDFS-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-13023:
-
Description:
Fails with the following exception.
{code}
2018-01-10 01:15:40,517 INFO server.JournalNodeSyncer
(JournalNodeSyncer.java:syncWithJournalAtIndex(235)) - Syncing Journal
/0.0.0.0:8485 with xxx, journal id: mycluster
2018-01-10 01:15:40,583 ERROR server.JournalNodeSyncer
(JournalNodeSyncer.java:syncWithJournalAtIndex(259)) - Could not sync with
Journal at xxx/xxx:8485
com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for
protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol:
this service is only accessible by nn/x...@example.com
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:242)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy16.getEditLogManifest(Unknown Source)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncWithJournalAtIndex(JournalNodeSyncer.java:254)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncJournals(JournalNodeSyncer.java:230)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.lambda$startSyncJournalsDaemon$0(JournalNodeSyncer.java:190)
at java.lang.Thread.run(Thread.java:748)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for
protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol:
this service is only accessible by nn/xxx
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1437)
at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
... 6 more
{code}
was:
2018-01-10 01:15:40,517 INFO server.JournalNodeSyncer
(JournalNodeSyncer.java:syncWithJournalAtIndex(235)) - Syncing Journal
/0.0.0.0:8485 with xxx, journal id: mycluster
2018-01-10 01:15:40,583 ERROR server.JournalNodeSyncer
(JournalNodeSyncer.java:syncWithJournalAtIndex(259)) - Could not sync with
Journal at xxx/xxx:8485
com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for
protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol:
this service is only accessible by nn/x...@example.com
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:242)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy16.getEditLogManifest(Unknown Source)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncWithJournalAtIndex(JournalNodeSyncer.java:254)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncJournals(JournalNodeSyncer.java:230)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.lambda$startSyncJournalsDaemon$0(JournalNodeSyncer.java:190)
at java.lang.Thread.run(Thread.java:748)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for
protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol:
this service is only accessible by nn/xxx
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1437)
at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
... 6 more
> Journal Sync not working on a kerberos cluster
> --
>
> Key: HDFS-13023
> URL: https://issues.apache.org/jira/browse/HDFS-13023
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDFS-13023.00.patch
>
>
> Fails with the following exception.
> {code}
> 2018-01-10 01:15:40,517 INFO server.JournalNodeSyncer
> (JournalNodeSyncer.java:syncWithJournalAtIndex(235)) - Syncing Journal
> /0.0.0.0:8485 with xxx, journal id: mycluster
> 2018-01-10 01:15:40,583 ERROR server.JournalNodeSyncer
> (JournalNodeSyncer.java:syncWithJournalAtIndex(259)) - Could not sync with
> Journal at xxx/xxx:8485
> com.google.protobuf.ServiceException:
>
[jira] [Updated] (HDFS-13023) Journal Sync not working on a kerberos cluster
[ https://issues.apache.org/jira/browse/HDFS-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13023: -- Status: Patch Available (was: In Progress) > Journal Sync not working on a kerberos cluster > -- > > Key: HDFS-13023 > URL: https://issues.apache.org/jira/browse/HDFS-13023 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13023.00.patch > > > 2018-01-10 01:15:40,517 INFO server.JournalNodeSyncer > (JournalNodeSyncer.java:syncWithJournalAtIndex(235)) - Syncing Journal > /0.0.0.0:8485 with xxx, journal id: mycluster > 2018-01-10 01:15:40,583 ERROR server.JournalNodeSyncer > (JournalNodeSyncer.java:syncWithJournalAtIndex(259)) - Could not sync with > Journal at xxx/xxx:8485 > com.google.protobuf.ServiceException: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): > User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for > protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol: > this service is only accessible by nn/x...@example.com > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:242) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy16.getEditLogManifest(Unknown Source) > at > org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncWithJournalAtIndex(JournalNodeSyncer.java:254) > at > org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncJournals(JournalNodeSyncer.java:230) > at > org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.lambda$startSyncJournalsDaemon$0(JournalNodeSyncer.java:190) > at java.lang.Thread.run(Thread.java:748) > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): > User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for > protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol: > this service is only accessible by nn/xxx > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491) > at org.apache.hadoop.ipc.Client.call(Client.java:1437) > at org.apache.hadoop.ipc.Client.call(Client.java:1347) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > ... 6 more -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13023) Journal Sync not working on a kerberos cluster
[ https://issues.apache.org/jira/browse/HDFS-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13023: -- Attachment: HDFS-13023.00.patch > Journal Sync not working on a kerberos cluster > -- > > Key: HDFS-13023 > URL: https://issues.apache.org/jira/browse/HDFS-13023 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13023.00.patch > > > 2018-01-10 01:15:40,517 INFO server.JournalNodeSyncer > (JournalNodeSyncer.java:syncWithJournalAtIndex(235)) - Syncing Journal > /0.0.0.0:8485 with xxx, journal id: mycluster > 2018-01-10 01:15:40,583 ERROR server.JournalNodeSyncer > (JournalNodeSyncer.java:syncWithJournalAtIndex(259)) - Could not sync with > Journal at xxx/xxx:8485 > com.google.protobuf.ServiceException: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): > User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for > protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol: > this service is only accessible by nn/x...@example.com > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:242) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy16.getEditLogManifest(Unknown Source) > at > org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncWithJournalAtIndex(JournalNodeSyncer.java:254) > at > org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncJournals(JournalNodeSyncer.java:230) > at > org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.lambda$startSyncJournalsDaemon$0(JournalNodeSyncer.java:190) > at java.lang.Thread.run(Thread.java:748) > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): > User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for > protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol: > this service is only accessible by nn/xxx > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491) > at org.apache.hadoop.ipc.Client.call(Client.java:1437) > at org.apache.hadoop.ipc.Client.call(Client.java:1347) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > ... 6 more -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-13023) Journal Sync not working on a kerberos cluster
[ https://issues.apache.org/jira/browse/HDFS-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-13023 started by Bharat Viswanadham. - > Journal Sync not working on a kerberos cluster > -- > > Key: HDFS-13023 > URL: https://issues.apache.org/jira/browse/HDFS-13023 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > 2018-01-10 01:15:40,517 INFO server.JournalNodeSyncer > (JournalNodeSyncer.java:syncWithJournalAtIndex(235)) - Syncing Journal > /0.0.0.0:8485 with xxx, journal id: mycluster > 2018-01-10 01:15:40,583 ERROR server.JournalNodeSyncer > (JournalNodeSyncer.java:syncWithJournalAtIndex(259)) - Could not sync with > Journal at xxx/xxx:8485 > com.google.protobuf.ServiceException: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): > User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for > protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol: > this service is only accessible by nn/x...@example.com > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:242) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy16.getEditLogManifest(Unknown Source) > at > org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncWithJournalAtIndex(JournalNodeSyncer.java:254) > at > org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncJournals(JournalNodeSyncer.java:230) > at > org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.lambda$startSyncJournalsDaemon$0(JournalNodeSyncer.java:190) > at java.lang.Thread.run(Thread.java:748) > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): > User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for > protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol: > this service is only accessible by nn/xxx > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491) > at org.apache.hadoop.ipc.Client.call(Client.java:1437) > at org.apache.hadoop.ipc.Client.call(Client.java:1347) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > ... 6 more -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13023) Journal Sync not working on a kerberos cluster
Bharat Viswanadham created HDFS-13023: - Summary: Journal Sync not working on a kerberos cluster Key: HDFS-13023 URL: https://issues.apache.org/jira/browse/HDFS-13023 Project: Hadoop HDFS Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham 2018-01-10 01:15:40,517 INFO server.JournalNodeSyncer (JournalNodeSyncer.java:syncWithJournalAtIndex(235)) - Syncing Journal /0.0.0.0:8485 with xxx, journal id: mycluster 2018-01-10 01:15:40,583 ERROR server.JournalNodeSyncer (JournalNodeSyncer.java:syncWithJournalAtIndex(259)) - Could not sync with Journal at xxx/xxx:8485 com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol: this service is only accessible by nn/x...@example.com at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:242) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy16.getEditLogManifest(Unknown Source) at org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncWithJournalAtIndex(JournalNodeSyncer.java:254) at org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncJournals(JournalNodeSyncer.java:230) at org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.lambda$startSyncJournalsDaemon$0(JournalNodeSyncer.java:190) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol: this service is only accessible by nn/xxx at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491) at org.apache.hadoop.ipc.Client.call(Client.java:1437) at org.apache.hadoop.ipc.Client.call(Client.java:1347) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) ... 6 more -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13017) Block Storage: implement simple iscsi discovery in jscsi server
[ https://issues.apache.org/jira/browse/HDFS-13017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327730#comment-16327730 ] Mukul Kumar Singh commented on HDFS-13017: -- Thanks for the patch [~elek]. The patch looks really good. Some minor comments 1) Please import the entire class "CBlockClientServerProtocolProtos.ListVolumesResponseProto" this will help in better formatting of functions. 2) CblockTargetServer.java:113, is never used, should we remove this line ? 3) Checkstyle issues 4) As Chen suggested, can a test be added to this, we have have a simple test to check for list volume after a volume is created. > Block Storage: implement simple iscsi discovery in jscsi server > --- > > Key: HDFS-13017 > URL: https://issues.apache.org/jira/browse/HDFS-13017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HDFS-13017-HDFS-7240.001.patch > > > The current jscsi server doesn't support iscsi discovery. > To use jscsi server as a kubernetes storage backend we need the discovery. > jScsi supports it we need just override a method and add an additional call > to the server protocl to get the list of the available cblocks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container
[ https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh reopened HDFS-12794: -- This patch has been reverted in HDFS-7240. Reopening it. > Ozone: Parallelize ChunkOutputSream Writes to container > --- > > Key: HDFS-12794 > URL: https://issues.apache.org/jira/browse/HDFS-12794 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-12794-HDFS-7240.001.patch, > HDFS-12794-HDFS-7240.002.patch, HDFS-12794-HDFS-7240.003.patch, > HDFS-12794-HDFS-7240.004.patch, HDFS-12794-HDFS-7240.005.patch, > HDFS-12794-HDFS-7240.006.patch, HDFS-12794-HDFS-7240.007.patch, > HDFS-12794-HDFS-7240.008.patch, HDFS-12794-HDFS-7240.009.patch > > > The chunkOutPutStream Write are sync in nature .Once one chunk of data gets > written, the next chunk write is blocked until the previous chunk is written > to the container. > The ChunkOutputWrite Stream writes should be made async and Close on the > OutputStream should ensure flushing of all dirty buffers to the container. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12051) Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327669#comment-16327669
]
Misha Dmitriev commented on HDFS-12051:
---
[~szetszwo] I hope I've addressed all your concerns?
> Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
>
[jira] [Commented] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container
[ https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327650#comment-16327650 ] Mukul Kumar Singh commented on HDFS-12794: -- This patch causes failure of tests TestCorona, TestKeys and TestKeysRatis. I will be reverting this patch. This patch currently depends upon HDFS-12986, there are also out of memory issues which I was able to see while debugging some of the test failures. > Ozone: Parallelize ChunkOutputSream Writes to container > --- > > Key: HDFS-12794 > URL: https://issues.apache.org/jira/browse/HDFS-12794 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-12794-HDFS-7240.001.patch, > HDFS-12794-HDFS-7240.002.patch, HDFS-12794-HDFS-7240.003.patch, > HDFS-12794-HDFS-7240.004.patch, HDFS-12794-HDFS-7240.005.patch, > HDFS-12794-HDFS-7240.006.patch, HDFS-12794-HDFS-7240.007.patch, > HDFS-12794-HDFS-7240.008.patch, HDFS-12794-HDFS-7240.009.patch > > > The chunkOutPutStream Write are sync in nature .Once one chunk of data gets > written, the next chunk write is blocked until the previous chunk is written > to the container. > The ChunkOutputWrite Stream writes should be made async and Close on the > OutputStream should ensure flushing of all dirty buffers to the container. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12993) Add an option to Balancer for excluding the paths
[
https://issues.apache.org/jira/browse/HDFS-12993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327487#comment-16327487
]
Ajay Kumar edited comment on HDFS-12993 at 1/16/18 6:09 PM:
{quote}{{getBlocks()}} is more expensive when there are many smaller blocks.
E.g. more blocks to go through to gather 10GB worth. Doing path exclusion in
such clusters can have a huge performance impact.
{quote}
[~kihwal], Seems blocks smaller than {{dfs.balancer.getBlocks.min-block-size}}
are excluded before this new path exclusion check happens. However i think we
can safely update check for minBlockSize to {{if (curBlock.getNumBytes() <=
getBlocksMinBlockSize)}}.
[~brahmareddy], Some comments:
# {{Balancer#excludedPaths}} doesn't set excludedPaths to
{{BalancerParameters.Builder }}{code} else if
("-excludedPaths".equalsIgnoreCase(args[i])) \{ checkArgument(++i <
args.length, "exclude paths is missing: args = " + Arrays.toString (args));
excludedPaths = args[i]; } {\code}
# Checkstyle issues in BalancerParameters(L109) ,Dispatcher (L107)
# Could you please add Javadoc for {{Dispatcher#getFilteredPaths,
BlockManager#excludeBlocks}}
# Can we move the check for excludePathPrefixes.isEmpty or null before we call
{
Unknown macro: \{BlockManager#excludeBlocks}
}
was (Author: ajayydv):
{quote}{{getBlocks()}} is more expensive when there are many smaller blocks.
E.g. more blocks to go through to gather 10GB worth. Doing path exclusion in
such clusters can have a huge performance impact.
{quote}
[~kihwal], Seems blocks smaller than \{{dfs.balancer.getBlocks.min-block-size}}
are excluded before this new path exclusion check happens. However i think we
can safely update check for minBlockSize to \{{if (curBlock.getNumBytes() <=
getBlocksMinBlockSize)}}.
[~brahmareddy], Some comments:
# {\{Balancer#excludedPaths}} doesn't set excludedPaths to
\{{BalancerParameters.Builder}}\{code}else if
("-excludedPaths".equalsIgnoreCase(args[i])) {
checkArgument(++i < args.length,
"exclude paths is missing: args = " + Arrays.toString
(args));
excludedPaths = args[i];
}\{code}
# Checkstyle issues in BalancerParameters(L109) ,Dispatcher (L107)
# Could you please add Javadoc for {{Dispatcher#getFilteredPaths,
BlockManager#excludeBlocks}}
# Can we move the check for excludePathPrefixes.isEmpty or null before we call
\{{BlockManager#excludeBlocks}}
> Add an option to Balancer for excluding the paths
> -
>
> Key: HDFS-12993
> URL: https://issues.apache.org/jira/browse/HDFS-12993
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer mover
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Major
> Attachments: HDFS-12993.patch
>
>
> *Usecase:* Users/Customers want to keep the important/frequently used
> blocks(where blocks are not written with favoured nodes and block pinning is
> not enabled) when balancer is triggered.This can be useful datacentric jobs
> which will run frequently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12993) Add an option to Balancer for excluding the paths
[
https://issues.apache.org/jira/browse/HDFS-12993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327487#comment-16327487
]
Ajay Kumar commented on HDFS-12993:
---
{quote}{{getBlocks()}} is more expensive when there are many smaller blocks.
E.g. more blocks to go through to gather 10GB worth. Doing path exclusion in
such clusters can have a huge performance impact.
{quote}
[~kihwal], Seems blocks smaller than \{{dfs.balancer.getBlocks.min-block-size}}
are excluded before this new path exclusion check happens. However i think we
can safely update check for minBlockSize to \{{if (curBlock.getNumBytes() <=
getBlocksMinBlockSize)}}.
[~brahmareddy], Some comments:
# {\{Balancer#excludedPaths}} doesn't set excludedPaths to
\{{BalancerParameters.Builder}}\{code}else if
("-excludedPaths".equalsIgnoreCase(args[i])) {
checkArgument(++i < args.length,
"exclude paths is missing: args = " + Arrays.toString
(args));
excludedPaths = args[i];
}\{code}
# Checkstyle issues in BalancerParameters(L109) ,Dispatcher (L107)
# Could you please add Javadoc for {{Dispatcher#getFilteredPaths,
BlockManager#excludeBlocks}}
# Can we move the check for excludePathPrefixes.isEmpty or null before we call
\{{BlockManager#excludeBlocks}}
> Add an option to Balancer for excluding the paths
> -
>
> Key: HDFS-12993
> URL: https://issues.apache.org/jira/browse/HDFS-12993
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer mover
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Major
> Attachments: HDFS-12993.patch
>
>
> *Usecase:* Users/Customers want to keep the important/frequently used
> blocks(where blocks are not written with favoured nodes and block pinning is
> not enabled) when balancer is triggered.This can be useful datacentric jobs
> which will run frequently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12973) RBF: Document global quota supporting in federation
[
https://issues.apache.org/jira/browse/HDFS-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327420#comment-16327420
]
Íñigo Goiri commented on HDFS-12973:
Thanks [~linyiqun] for the patch. A few comments:
* Should we add a link from/to {{HdfsQuotaAdminGuide.md}}?
* In {{HDFSRouterFederation.md}}:
** We can avoid {{Router has another one optional role: Quota Manager.}} as
this is the section itself.
** The text shouldn't use {{Now}} as this text will stay there in future
releases.
** It should be {{control this *at* mount table level}}
** {{For performance reasons, the Router caches the quota usage and updates it
periodically}}.
** {{supports setting quotas for specified mount table entries}}
** {{The above command}}
** For the update interval, we should mention it supports periods and not only
milliseconds.
> RBF: Document global quota supporting in federation
> ---
>
> Key: HDFS-12973
> URL: https://issues.apache.org/jira/browse/HDFS-12973
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDFS-12973.001.patch
>
>
> Document global quota supporting in federation
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12522) Ozone: Remove the Priority Queues used in the Container State Manager
[ https://issues.apache.org/jira/browse/HDFS-12522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327373#comment-16327373 ] Anu Engineer commented on HDFS-12522: - [~elek] I have a patch for this, Would you mind If I assigned it back to me? > Ozone: Remove the Priority Queues used in the Container State Manager > - > > Key: HDFS-12522 > URL: https://issues.apache.org/jira/browse/HDFS-12522 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Major > > During code review of HDFS-12387, it was suggested that we remove the > priority queues that was used in ContainerStateManager. This JIRA tracks that > issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12522) Ozone: Remove the Priority Queues used in the Container State Manager
[ https://issues.apache.org/jira/browse/HDFS-12522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer reassigned HDFS-12522: --- Assignee: Anu Engineer (was: Elek, Marton) > Ozone: Remove the Priority Queues used in the Container State Manager > - > > Key: HDFS-12522 > URL: https://issues.apache.org/jira/browse/HDFS-12522 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Major > > During code review of HDFS-12387, it was suggested that we remove the > priority queues that was used in ContainerStateManager. This JIRA tracks that > issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13003) Access time on dir changed via setTimes() should be stored in fsimage
[ https://issues.apache.org/jira/browse/HDFS-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318339#comment-16318339 ] Brahma Reddy Battula edited comment on HDFS-13003 at 1/16/18 2:40 PM: -- [~sree88] thanks for reporting this. Nice finding! can you add one testcase for same..? Added you as HDFS contributor,from now onwards you can assign issues your self. Please go through [How to contribute|https://wiki.apache.org/hadoop/HowToContribute] for more details. was (Author: brahmareddy): [~sree88] thanks for reporting this. it make sense to give "0" as other fileds are zero. Added you as HDFS contributor,from now onwards you can assign issues your self. Please go through [How to contribute|https://wiki.apache.org/hadoop/HowToContribute] for more details. > Access time on dir changed via setTimes() should be stored in fsimage > - > > Key: HDFS-13003 > URL: https://issues.apache.org/jira/browse/HDFS-13003 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Sreejith MV >Assignee: Sreejith MV >Priority: Minor > Attachments: HDFS-13003.patch > > > Access time for a directory can be modified with > DistributedFileSystem.setTimes(). > But this changed access time is not stored in the fsimage. > After restart of namenode, it will be lost and reset as zero. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8693) refreshNamenodes does not support adding a new standby to a running DN
[
https://issues.apache.org/jira/browse/HDFS-8693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-8693:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.4
Status: Resolved (was: Patch Available)
Committed to {{trunk}}, {{branch-3.0}} , {{branch-2}} , {{branch-2.9}} and
{{branch-2.8}}.
Thanks [~ajithshetty] for contributing and thanks to [~vinayrpet] for
additional review and thanks others for great discussion. [~john.jian.fang]
thanks for reporting.
> refreshNamenodes does not support adding a new standby to a running DN
> --
>
> Key: HDFS-8693
> URL: https://issues.apache.org/jira/browse/HDFS-8693
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, ha
>Affects Versions: 2.6.0
>Reporter: Jian Fang
>Assignee: Ajith S
>Priority: Critical
> Fix For: 2.8.4
>
> Attachments: HDFS-8693.02.patch, HDFS-8693.03.patch, HDFS-8693.1.patch
>
>
> I tried to run the following command on a Hadoop 2.6.0 cluster with HA
> support
> $ hdfs dfsadmin -refreshNamenodes datanode-host:port
> to refresh name nodes on data nodes after I replaced one name node with a new
> one so that I don't need to restart the data nodes. However, I got the
> following error:
> refreshNamenodes: HA does not currently support adding a new standby to a
> running DN. Please do a rolling restart of DNs to reconfigure the list of NNs.
> I checked the 2.6.0 code and the error was thrown by the following code
> snippet, which led me to this JIRA.
> void refreshNNList(ArrayList addrs) throws IOException {
> Set oldAddrs = Sets.newHashSet();
> for (BPServiceActor actor : bpServices)
> { oldAddrs.add(actor.getNNSocketAddress()); }
> Set newAddrs = Sets.newHashSet(addrs);
> if (!Sets.symmetricDifference(oldAddrs, newAddrs).isEmpty())
> { // Keep things simple for now -- we can implement this at a later date.
> throw new IOException( "HA does not currently support adding a new standby to
> a running DN. " + "Please do a rolling restart of DNs to reconfigure the list
> of NNs."); }
> }
> Looks like this the refreshNameNodes command is an uncompleted feature.
> Unfortunately, the new name node on a replacement is critical for auto
> provisioning a hadoop cluster with HDFS HA support. Without this support, the
> HA feature could not really be used. I also observed that the new standby
> name node on the replacement instance could stuck in safe mode because no
> data nodes check in with it. Even with a rolling restart, it may take quite
> some time to restart all data nodes if we have a big cluster, for example,
> with 4000 data nodes, let alone restarting DN is way too intrusive and it is
> not a preferable operation in production. It also increases the chance for a
> double failure because the standby name node is not really ready for a
> failover in the case that the current active name node fails.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13003) Access time on dir changed via setTimes() should be stored in fsimage
[ https://issues.apache.org/jira/browse/HDFS-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreejith MV updated HDFS-13003: --- Summary: Access time on dir changed via setTimes() should be stored in fsimage (was: Access time should be considered when setTimes is called on a dir) > Access time on dir changed via setTimes() should be stored in fsimage > - > > Key: HDFS-13003 > URL: https://issues.apache.org/jira/browse/HDFS-13003 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Sreejith MV >Assignee: Sreejith MV >Priority: Minor > Attachments: HDFS-13003.patch > > > Access time for a directory can be modified with > DistributedFileSystem.setTimes(). > But this changed access time is not stored in the fsimage. > After restart of namenode, it will be lost and reset as zero. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12989) Ozone: Container : Add key versioning support-4
[
https://issues.apache.org/jira/browse/HDFS-12989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-12989:
-
Issue Type: Sub-task (was: Improvement)
Parent: HDFS-7240
> Ozone: Container : Add key versioning support-4
> ---
>
> Key: HDFS-12989
> URL: https://issues.apache.org/jira/browse/HDFS-12989
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Attachments: HDFS-12989-HDFS-7240.001.patch
>
>
> After HDFS-12925 and HDFS-12954 get added, every key write call will generate
> a new version, and we will be able to read any specific version of a key.
> This JIRA adds a new newKeyReader API {{newKeyReaderWithVersion}} to
> {{StorageHandler}}. This method takes an extra version field such that the
> caller can read any older version. This JIRA also adds all the other changes
> needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13003) Access time should be considered when setTimes is called on a dir
[ https://issues.apache.org/jira/browse/HDFS-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreejith MV updated HDFS-13003: --- Summary: Access time should be considered when setTimes is called on a dir (was: [OIV] Irrelevant access time for the dirs when run with Delimited option ) Description: Access time for a directory can be modified with DistributedFileSystem.setTimes(). But this changed access time is not stored in the fsimage. After restart of namenode, it will be lost and reset as zero. was: *#hdfs oiv -i fsimage_0018779 -p Delimited -o test* 2018-01-09 18:37:13,218 INFO offlineImageViewer.PBImageTextWriter: Loading string table 2018-01-09 18:37:13,227 INFO offlineImageViewer.FSImageHandler: Loading 2 strings 2018-01-09 18:37:13,229 INFO offlineImageViewer.PBImageTextWriter: Loading inode references 2018-01-09 18:37:13,229 INFO offlineImageViewer.FSImageHandler: Loading inode references 2018-01-09 18:37:13,235 INFO offlineImageViewer.FSImageHandler: Loaded 0 inode references 2018-01-09 18:37:13,236 INFO offlineImageViewer.PBImageTextWriter: Loading directories 2018-01-09 18:37:13,240 INFO offlineImageViewer.PBImageTextWriter: Loading directories in INode section. 2018-01-09 18:37:13,263 INFO offlineImageViewer.PBImageTextWriter: Found 1 directories in INode section. 2018-01-09 18:37:13,264 INFO offlineImageViewer.PBImageTextWriter: Finished loading directories in 25ms 2018-01-09 18:37:13,264 INFO offlineImageViewer.PBImageTextWriter: Loading INode directory section. 2018-01-09 18:37:13,267 INFO offlineImageViewer.PBImageTextWriter: Scanned 0 INode directories to build namespace. 2018-01-09 18:37:13,267 INFO offlineImageViewer.PBImageTextWriter: Finished loading INode directory section in 3ms *#cat test* PathReplication ModificationTime*AccessTime* PreferredBlockSize BlocksCount FileSizeNSQUOTA DSQUOTA Permission UserNameGroupName /dirSetExpr40 2017-12-28 23:39*1970-01-01 08:00*0 0 0 -1 -1 drwxr-xr-x root hadoop /dirSetExpr60 2017-12-28 23:39*1970-01-01 08:00*0 0 0 -1 -1 drwxr-xr-x root hadoop /dirSetExpr30 2017-12-28 23:39*1970-01-01 08:00*0 0 0 -1 -1 drwxr-xr-x root hadoop /dirSetExpr20 2017-12-28 23:39*1970-01-01 08:00 * 0 0 0 -1 -1 drwxr-xr-x root hadoop /dirSetExpr70 2017-12-28 23:39*1970-01-01 08:00*0 0 0 -1 -1 drwxr-xr-x root hadoop /dirSetExpr5/fsumm261970.txt3 2017-12-28 22:102017-12-28 22:101048576 1 38 0 0 -rw-r--r-- root hadoop /dirSetExpr6/fsumm261970.txt3 2017-12-28 22:102017-12-28 22:101048576 1 38 0 0 -rw-r--r-- root hadoop > Access time should be considered when setTimes is called on a dir > - > > Key: HDFS-13003 > URL: https://issues.apache.org/jira/browse/HDFS-13003 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Sreejith MV >Assignee: Sreejith MV >Priority: Minor > Attachments: HDFS-13003.patch > > > Access time for a directory can be modified with > DistributedFileSystem.setTimes(). > But this changed access time is not stored in the fsimage. > After restart of namenode, it will be lost and reset as zero. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-13003) [OIV] Irrelevant access time for the dirs when run with Delimited option
[ https://issues.apache.org/jira/browse/HDFS-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreejith MV updated HDFS-13003: --- Comment: was deleted (was: Thanks [~brahmareddy] for the comments. I have attached the initial patch. Please review.) > [OIV] Irrelevant access time for the dirs when run with Delimited option > -- > > Key: HDFS-13003 > URL: https://issues.apache.org/jira/browse/HDFS-13003 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Sreejith MV >Assignee: Sreejith MV >Priority: Minor > Attachments: HDFS-13003.patch > > > *#hdfs oiv -i fsimage_0018779 -p Delimited -o test* > 2018-01-09 18:37:13,218 INFO offlineImageViewer.PBImageTextWriter: Loading > string table > 2018-01-09 18:37:13,227 INFO offlineImageViewer.FSImageHandler: Loading 2 > strings > 2018-01-09 18:37:13,229 INFO offlineImageViewer.PBImageTextWriter: Loading > inode references > 2018-01-09 18:37:13,229 INFO offlineImageViewer.FSImageHandler: Loading inode > references > 2018-01-09 18:37:13,235 INFO offlineImageViewer.FSImageHandler: Loaded 0 > inode references > 2018-01-09 18:37:13,236 INFO offlineImageViewer.PBImageTextWriter: Loading > directories > 2018-01-09 18:37:13,240 INFO offlineImageViewer.PBImageTextWriter: Loading > directories in INode section. > 2018-01-09 18:37:13,263 INFO offlineImageViewer.PBImageTextWriter: Found 1 > directories in INode section. > 2018-01-09 18:37:13,264 INFO offlineImageViewer.PBImageTextWriter: Finished > loading directories in 25ms > 2018-01-09 18:37:13,264 INFO offlineImageViewer.PBImageTextWriter: Loading > INode directory section. > 2018-01-09 18:37:13,267 INFO offlineImageViewer.PBImageTextWriter: Scanned 0 > INode directories to build namespace. > 2018-01-09 18:37:13,267 INFO offlineImageViewer.PBImageTextWriter: Finished > loading INode directory section in 3ms > *#cat test* > PathReplication ModificationTime*AccessTime* > PreferredBlockSize BlocksCount FileSizeNSQUOTA DSQUOTA > Permission UserNameGroupName > /dirSetExpr40 2017-12-28 23:39*1970-01-01 08:00*0 > 0 0 -1 -1 drwxr-xr-x root hadoop > /dirSetExpr60 2017-12-28 23:39*1970-01-01 08:00*0 > 0 0 -1 -1 drwxr-xr-x root hadoop > /dirSetExpr30 2017-12-28 23:39*1970-01-01 08:00*0 > 0 0 -1 -1 drwxr-xr-x root hadoop > /dirSetExpr20 2017-12-28 23:39*1970-01-01 08:00 * 0 > 0 0 -1 -1 drwxr-xr-x root hadoop > /dirSetExpr70 2017-12-28 23:39*1970-01-01 08:00*0 > 0 0 -1 -1 drwxr-xr-x root hadoop > /dirSetExpr5/fsumm261970.txt3 2017-12-28 22:102017-12-28 > 22:101048576 1 38 0 0 -rw-r--r-- root > hadoop > /dirSetExpr6/fsumm261970.txt3 2017-12-28 22:102017-12-28 > 22:101048576 1 38 0 0 -rw-r--r-- root > hadoop -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13003) [OIV] Irrelevant access time for the dirs when run with Delimited option
[ https://issues.apache.org/jira/browse/HDFS-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327077#comment-16327077 ] Sreejith MV commented on HDFS-13003: Thanks [~brahmareddy] for the comments. I have attached the initial patch. Please review. > [OIV] Irrelevant access time for the dirs when run with Delimited option > -- > > Key: HDFS-13003 > URL: https://issues.apache.org/jira/browse/HDFS-13003 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Sreejith MV >Assignee: Sreejith MV >Priority: Minor > Attachments: HDFS-13003.patch > > > *#hdfs oiv -i fsimage_0018779 -p Delimited -o test* > 2018-01-09 18:37:13,218 INFO offlineImageViewer.PBImageTextWriter: Loading > string table > 2018-01-09 18:37:13,227 INFO offlineImageViewer.FSImageHandler: Loading 2 > strings > 2018-01-09 18:37:13,229 INFO offlineImageViewer.PBImageTextWriter: Loading > inode references > 2018-01-09 18:37:13,229 INFO offlineImageViewer.FSImageHandler: Loading inode > references > 2018-01-09 18:37:13,235 INFO offlineImageViewer.FSImageHandler: Loaded 0 > inode references > 2018-01-09 18:37:13,236 INFO offlineImageViewer.PBImageTextWriter: Loading > directories > 2018-01-09 18:37:13,240 INFO offlineImageViewer.PBImageTextWriter: Loading > directories in INode section. > 2018-01-09 18:37:13,263 INFO offlineImageViewer.PBImageTextWriter: Found 1 > directories in INode section. > 2018-01-09 18:37:13,264 INFO offlineImageViewer.PBImageTextWriter: Finished > loading directories in 25ms > 2018-01-09 18:37:13,264 INFO offlineImageViewer.PBImageTextWriter: Loading > INode directory section. > 2018-01-09 18:37:13,267 INFO offlineImageViewer.PBImageTextWriter: Scanned 0 > INode directories to build namespace. > 2018-01-09 18:37:13,267 INFO offlineImageViewer.PBImageTextWriter: Finished > loading INode directory section in 3ms > *#cat test* > PathReplication ModificationTime*AccessTime* > PreferredBlockSize BlocksCount FileSizeNSQUOTA DSQUOTA > Permission UserNameGroupName > /dirSetExpr40 2017-12-28 23:39*1970-01-01 08:00*0 > 0 0 -1 -1 drwxr-xr-x root hadoop > /dirSetExpr60 2017-12-28 23:39*1970-01-01 08:00*0 > 0 0 -1 -1 drwxr-xr-x root hadoop > /dirSetExpr30 2017-12-28 23:39*1970-01-01 08:00*0 > 0 0 -1 -1 drwxr-xr-x root hadoop > /dirSetExpr20 2017-12-28 23:39*1970-01-01 08:00 * 0 > 0 0 -1 -1 drwxr-xr-x root hadoop > /dirSetExpr70 2017-12-28 23:39*1970-01-01 08:00*0 > 0 0 -1 -1 drwxr-xr-x root hadoop > /dirSetExpr5/fsumm261970.txt3 2017-12-28 22:102017-12-28 > 22:101048576 1 38 0 0 -rw-r--r-- root > hadoop > /dirSetExpr6/fsumm261970.txt3 2017-12-28 22:102017-12-28 > 22:101048576 1 38 0 0 -rw-r--r-- root > hadoop -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13003) [OIV] Irrelevant access time for the dirs when run with Delimited option
[ https://issues.apache.org/jira/browse/HDFS-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreejith MV updated HDFS-13003: --- Attachment: HDFS-13003.patch > [OIV] Irrelevant access time for the dirs when run with Delimited option > -- > > Key: HDFS-13003 > URL: https://issues.apache.org/jira/browse/HDFS-13003 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Sreejith MV >Assignee: Sreejith MV >Priority: Minor > Attachments: HDFS-13003.patch > > > *#hdfs oiv -i fsimage_0018779 -p Delimited -o test* > 2018-01-09 18:37:13,218 INFO offlineImageViewer.PBImageTextWriter: Loading > string table > 2018-01-09 18:37:13,227 INFO offlineImageViewer.FSImageHandler: Loading 2 > strings > 2018-01-09 18:37:13,229 INFO offlineImageViewer.PBImageTextWriter: Loading > inode references > 2018-01-09 18:37:13,229 INFO offlineImageViewer.FSImageHandler: Loading inode > references > 2018-01-09 18:37:13,235 INFO offlineImageViewer.FSImageHandler: Loaded 0 > inode references > 2018-01-09 18:37:13,236 INFO offlineImageViewer.PBImageTextWriter: Loading > directories > 2018-01-09 18:37:13,240 INFO offlineImageViewer.PBImageTextWriter: Loading > directories in INode section. > 2018-01-09 18:37:13,263 INFO offlineImageViewer.PBImageTextWriter: Found 1 > directories in INode section. > 2018-01-09 18:37:13,264 INFO offlineImageViewer.PBImageTextWriter: Finished > loading directories in 25ms > 2018-01-09 18:37:13,264 INFO offlineImageViewer.PBImageTextWriter: Loading > INode directory section. > 2018-01-09 18:37:13,267 INFO offlineImageViewer.PBImageTextWriter: Scanned 0 > INode directories to build namespace. > 2018-01-09 18:37:13,267 INFO offlineImageViewer.PBImageTextWriter: Finished > loading INode directory section in 3ms > *#cat test* > PathReplication ModificationTime*AccessTime* > PreferredBlockSize BlocksCount FileSizeNSQUOTA DSQUOTA > Permission UserNameGroupName > /dirSetExpr40 2017-12-28 23:39*1970-01-01 08:00*0 > 0 0 -1 -1 drwxr-xr-x root hadoop > /dirSetExpr60 2017-12-28 23:39*1970-01-01 08:00*0 > 0 0 -1 -1 drwxr-xr-x root hadoop > /dirSetExpr30 2017-12-28 23:39*1970-01-01 08:00*0 > 0 0 -1 -1 drwxr-xr-x root hadoop > /dirSetExpr20 2017-12-28 23:39*1970-01-01 08:00 * 0 > 0 0 -1 -1 drwxr-xr-x root hadoop > /dirSetExpr70 2017-12-28 23:39*1970-01-01 08:00*0 > 0 0 -1 -1 drwxr-xr-x root hadoop > /dirSetExpr5/fsumm261970.txt3 2017-12-28 22:102017-12-28 > 22:101048576 1 38 0 0 -rw-r--r-- root > hadoop > /dirSetExpr6/fsumm261970.txt3 2017-12-28 22:102017-12-28 > 22:101048576 1 38 0 0 -rw-r--r-- root > hadoop -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12911) [SPS]: Modularize the SPS code and expose necessary interfaces for external/internal implementations.
[
https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327048#comment-16327048
]
genericqa commented on HDFS-12911:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
4s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-10285 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
40s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 31s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
53s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-10285 has 1
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} HDFS-10285 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 21 new + 334 unchanged - 0 fixed = 355 total (was 334) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 46s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}107m 26s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}172m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12911 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905895/HDFS-12911-HDFS-10285-03.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 6c854bf0edcc 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-10285 / ed60479 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/22669/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/22669/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| whitespace |
[jira] [Commented] (HDFS-8693) refreshNamenodes does not support adding a new standby to a running DN
[
https://issues.apache.org/jira/browse/HDFS-8693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327036#comment-16327036
]
Hudson commented on HDFS-8693:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13500 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13500/])
HDFS-8693. refreshNamenodes does not support adding a new standby to a (brahma:
rev 880b9d24ff7b5f350ec99bac9b0862009460b486)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
> refreshNamenodes does not support adding a new standby to a running DN
> --
>
> Key: HDFS-8693
> URL: https://issues.apache.org/jira/browse/HDFS-8693
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, ha
>Affects Versions: 2.6.0
>Reporter: Jian Fang
>Assignee: Ajith S
>Priority: Critical
> Attachments: HDFS-8693.02.patch, HDFS-8693.03.patch, HDFS-8693.1.patch
>
>
> I tried to run the following command on a Hadoop 2.6.0 cluster with HA
> support
> $ hdfs dfsadmin -refreshNamenodes datanode-host:port
> to refresh name nodes on data nodes after I replaced one name node with a new
> one so that I don't need to restart the data nodes. However, I got the
> following error:
> refreshNamenodes: HA does not currently support adding a new standby to a
> running DN. Please do a rolling restart of DNs to reconfigure the list of NNs.
> I checked the 2.6.0 code and the error was thrown by the following code
> snippet, which led me to this JIRA.
> void refreshNNList(ArrayList addrs) throws IOException {
> Set oldAddrs = Sets.newHashSet();
> for (BPServiceActor actor : bpServices)
> { oldAddrs.add(actor.getNNSocketAddress()); }
> Set newAddrs = Sets.newHashSet(addrs);
> if (!Sets.symmetricDifference(oldAddrs, newAddrs).isEmpty())
> { // Keep things simple for now -- we can implement this at a later date.
> throw new IOException( "HA does not currently support adding a new standby to
> a running DN. " + "Please do a rolling restart of DNs to reconfigure the list
> of NNs."); }
> }
> Looks like this the refreshNameNodes command is an uncompleted feature.
> Unfortunately, the new name node on a replacement is critical for auto
> provisioning a hadoop cluster with HDFS HA support. Without this support, the
> HA feature could not really be used. I also observed that the new standby
> name node on the replacement instance could stuck in safe mode because no
> data nodes check in with it. Even with a rolling restart, it may take quite
> some time to restart all data nodes if we have a big cluster, for example,
> with 4000 data nodes, let alone restarting DN is way too intrusive and it is
> not a preferable operation in production. It also increases the chance for a
> double failure because the standby name node is not really ready for a
> failover in the case that the current active name node fails.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up
[
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327022#comment-16327022
]
Brahma Reddy Battula commented on HDFS-12935:
-
Overall patch lgtm,apart from the following.
bq.I think \{{listOpenFiles }} is not needed to be fixed as the code following.
Not all namenodes are called in \{{listOpenFiles }} like other commands.
Yes,But when the this node is down immediately/tranistion to standby which will
not retried.So please handle as {{setbalancerwidth}}.
> Get ambiguous result for DFSAdmin command in HA mode when only one namenode
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.9.0, 3.0.0-beta1, 3.0.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Major
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch,
> HDFS-12935.004.patch, HDFS-12935.005.patch, HDFS-12935.006-branch.2.patch,
> HDFS-12935.006.patch, HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When
> considering the following two occasions:
> (1)nn1 up and nn2 down
> (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin
> commands will have ambiguous results. The commands can be send successfully
> to the up namenode and are always functionally useful only when nn1 is up
> regardless of exception (IOException when connecting to the down namenode
> nn2). If only nn2 is up, the commands have no use at all and only exception
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to
> set balancer bandwidth value for datanodes as an example. It works and all
> the datanodes can get the setting values only when nn1 is up. If only nn2 is
> up, the command throws exception directly and no datanode get the bandwidth
> setting. Approximately ten DFSAdmin commands use the similar logical process
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to
> jiangjianfei02:9820 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to
> jiangjianfei01:9820 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]#
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12654) APPEND API call is different in HTTPFS and NameNode REST
[
https://issues.apache.org/jira/browse/HDFS-12654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327000#comment-16327000
]
Masatake Iwasaki commented on HDFS-12654:
-
Hi [~Nuke], could you explain the way to reproduce the issue? I got the same
result for webhdfs and httpfs when I tested this on 3.1.0-SNAPSHOT.
{noformat}
$ curl -X POST -T README.txt -L -i
"http://localhost:9870/webhdfs/v1/tmp/README.txt?op=APPEND;
HTTP/1.1 100 Continue
HTTP/1.1 404 Not Found
Date: Tue, 16 Jan 2018 10:41:24 GMT
Cache-Control: no-cache
Expires: Tue, 16 Jan 2018 10:41:24 GMT
Date: Tue, 16 Jan 2018 10:41:24 GMT
Pragma: no-cache
X-FRAME-OPTIONS: SAMEORIGIN
Content-Type: application/json
Transfer-Encoding: chunked
{"RemoteException":{"exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException","message":"File
/tmp/README.txt not found."}}[centos@ip-172-32-1-195 hadoop-3.1.0-SNAPSHOT]$
{noformat}
{noformat}
$ curl -X POST -i --header "Content-Type:application/octet-stream"
--data-binary @README.txt
'http://localhost:14000/webhdfs/v1/tmp/README.txt?op=APPEND=centos=true'
HTTP/1.1 100 Continue
HTTP/1.1 404 Not Found
Date: Tue, 16 Jan 2018 10:42:17 GMT
Cache-Control: no-cache
Expires: Tue, 16 Jan 2018 10:42:17 GMT
Date: Tue, 16 Jan 2018 10:42:17 GMT
Pragma: no-cache
Set-Cookie:
hadoop.auth="u=centos=centos=simple-dt=1516135337720=ZDSoknG+/x/a6XnGLAVUyUBs6vE=";
Path=/; HttpOnly
Content-Type: application/json
Transfer-Encoding: chunked
{"RemoteException":{"message":"Failed to append to non-existent file
\/tmp\/README.txt for client
127.0.0.1","exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException"}}
{noformat}
> APPEND API call is different in HTTPFS and NameNode REST
>
>
> Key: HDFS-12654
> URL: https://issues.apache.org/jira/browse/HDFS-12654
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, httpfs, namenode
>Affects Versions: 2.6.0, 2.7.0, 2.8.0, 3.0.0-beta1
>Reporter: Andras Czesznak
>Priority: Major
>
> The APPEND REST API call behaves differently in the NameNode REST and the
> HTTPFS codes. The NameNode version creates the target file the new data being
> appended to if it does not exist at the time of the call issued. The HTTPFS
> version assumes the target file exists when APPEND is called and can append
> only the new data but does not create the target file it doesn't exist.
> The two implementations should be standardized, preferably the HTTPFS version
> should be modified to execute an implicit CREATE if the target file does not
> exist.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12973) RBF: Document global quota supporting in federation
[ https://issues.apache.org/jira/browse/HDFS-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-12973: - Attachment: HDFS-12973.001.patch > RBF: Document global quota supporting in federation > --- > > Key: HDFS-12973 > URL: https://issues.apache.org/jira/browse/HDFS-12973 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Major > Attachments: HDFS-12973.001.patch > > > Document global quota supporting in federation -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12973) RBF: Document global quota supporting in federation
[ https://issues.apache.org/jira/browse/HDFS-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-12973: - Attachment: (was: HDFS-12973.001.patch) > RBF: Document global quota supporting in federation > --- > > Key: HDFS-12973 > URL: https://issues.apache.org/jira/browse/HDFS-12973 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Major > Attachments: HDFS-12973.001.patch > > > Document global quota supporting in federation -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12462) Erasure coding policy extra options should be sorted by key value
[ https://issues.apache.org/jira/browse/HDFS-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326865#comment-16326865 ] liaoyuxiangqin commented on HDFS-12462: --- Thanks [~Sammi] for detailed explanation, i think all of extra options compare between fsimage and memory allways on the ECSchema level, so i sort extra options by used TreeMap's wrapped in ECSchema constructor. thanks! > Erasure coding policy extra options should be sorted by key value > - > > Key: HDFS-12462 > URL: https://issues.apache.org/jira/browse/HDFS-12462 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: liaoyuxiangqin >Priority: Major > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12462.001.patch > > > To make sure the serialized fsimage and editlog binary equal, Erasure coding > policy extra options should be sorted by key value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-12462) Erasure coding policy extra options should be sorted by key value
[ https://issues.apache.org/jira/browse/HDFS-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liaoyuxiangqin updated HDFS-12462: -- Comment: was deleted (was: Thanks [~Sammi] for detailed explanation, I have read the save ec policy to fsimg related code and process processing, but i still have a doubt about What scenario needs to compare between two fsImage or between fsImage with editlog, thanks.) > Erasure coding policy extra options should be sorted by key value > - > > Key: HDFS-12462 > URL: https://issues.apache.org/jira/browse/HDFS-12462 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: liaoyuxiangqin >Priority: Major > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12462.001.patch > > > To make sure the serialized fsimage and editlog binary equal, Erasure coding > policy extra options should be sorted by key value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12462) Erasure coding policy extra options should be sorted by key value
[ https://issues.apache.org/jira/browse/HDFS-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liaoyuxiangqin updated HDFS-12462: -- Assignee: liaoyuxiangqin Status: Patch Available (was: Open) > Erasure coding policy extra options should be sorted by key value > - > > Key: HDFS-12462 > URL: https://issues.apache.org/jira/browse/HDFS-12462 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: liaoyuxiangqin >Priority: Major > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12462.001.patch > > > To make sure the serialized fsimage and editlog binary equal, Erasure coding > policy extra options should be sorted by key value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12462) Erasure coding policy extra options should be sorted by key value
[ https://issues.apache.org/jira/browse/HDFS-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liaoyuxiangqin updated HDFS-12462: -- Attachment: HDFS-12462.001.patch > Erasure coding policy extra options should be sorted by key value > - > > Key: HDFS-12462 > URL: https://issues.apache.org/jira/browse/HDFS-12462 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Priority: Major > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12462.001.patch > > > To make sure the serialized fsimage and editlog binary equal, Erasure coding > policy extra options should be sorted by key value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
