[jira] [Created] (HDFS-13905) HDFS NameNode jmx information inaccuracy
dengsc created HDFS-13905:
-
Summary: HDFS NameNode jmx information inaccuracy
Key: HDFS-13905
URL: https://issues.apache.org/jira/browse/HDFS-13905
Project: Hadoop HDFS
Issue Type: Bug
Components: nn
Reporter: dengsc
The information obtained using Namenode jmx does not match the actual situation。
jmx json data:
{code:java}
// http://{namenode_ip}:50070/jmx/?qry=Hadoop:service=NameNode,name=BlockStats
{
"beans" : [ {
"name" : "Hadoop:service=NameNode,name=BlockStats",
"modelerType" :
"org.apache.hadoop.hdfs.server.blockmanagement.BlockManager",
"StorageTypeStats" : [ {
"key" : "ARCHIVE",
"value" : {
"blockPoolUsed" : 10411907414335294,
"capacityRemaining" : 1435938815980942,
"capacityTotal" : 11851673443205120,
"capacityUsed" : 10411907414335294,
"nodesInService" : 111
}
}, {
"key" : "DISK",
"value" : {
"blockPoolUsed" : 16789505803398474,
"capacityRemaining" : 3404376930515479,
"capacityTotal" : 20204114325184512,
"capacityUsed" : 16789505803398474,
"nodesInService" : 473
}
} ]
} ]
}
{code}
but ，in our cluster :
disk nodesInService = 462
archive nodesInService = 110
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13445) Hostname is not getting resolved when we pass IP in webhdfs request.
[
https://issues.apache.org/jira/browse/HDFS-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608755#comment-16608755
]
Hadoop QA commented on HDFS-13445:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 20s{color} | {color:orange} hadoop-common-project/hadoop-auth: The patch
generated 1 new + 27 unchanged - 0 fixed = 28 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 27s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
3s{color} | {color:green} hadoop-auth in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDFS-13445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918948/HDFS-13445.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 630e716075a7 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / eef3baf |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/25015/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-auth.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/25015/artifact/out/whitespace-tabs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25015/testReport/ |
| Max. process+thread count | 409 (vs. ulimit of 1) |
|
[jira] [Commented] (HDFS-13243) Get CorruptBlock because of calling close and sync in same time
[
https://issues.apache.org/jira/browse/HDFS-13243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608737#comment-16608737
]
Hadoop QA commented on HDFS-13243:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-13243 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-13243 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918525/HDFS-13243-v6.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25016/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Get CorruptBlock because of calling close and sync in same time
> ---
>
> Key: HDFS-13243
> URL: https://issues.apache.org/jira/browse/HDFS-13243
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2, 3.2.0
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>Priority: Critical
> Attachments: HDFS-13243-v1.patch, HDFS-13243-v2.patch,
> HDFS-13243-v3.patch, HDFS-13243-v4.patch, HDFS-13243-v5.patch,
> HDFS-13243-v6.patch
>
>
> HDFS File might get broken because of corrupt block(s) that could be produced
> by calling close and sync in the same time.
> When calling close was not successful, UCBlock status would change to
> COMMITTED, and if a sync request gets popped from queue and processed, sync
> operation would change the last block length.
> After that, DataNode would report all received block to NameNode, and will
> check Block length of all COMMITTED Blocks. But the block length was already
> different between recorded in NameNode memory and reported by DataNode, and
> consequently, the last block is marked as corruptted because of inconsistent
> length.
>
> {panel:title=Log in my hdfs}
> 2018-03-05 04:05:39,261 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> allocate blk_1085498930_11758129\{UCState=UNDER_CONSTRUCTION,
> truncateBlock=null, primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-32c7e479-3845-4a44-adf1-831edec7506b:NORMAL:10.0.0.219:50010|RBW],
>
> ReplicaUC[[DISK]DS-a9a5d653-c049-463d-8e4a-d1f0dc14409c:NORMAL:10.0.0.220:50010|RBW],
>
> ReplicaUC[[DISK]DS-f2b7c04a-b724-4c69-abbf-d2e416f70706:NORMAL:10.0.0.218:50010|RBW]]}
> for
> /hbase/WALs/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com,16020,1519845790686/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com%2C16020%2C1519845790686.default.1520193926515
> 2018-03-05 04:05:39,760 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> fsync:
> /hbase/WALs/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com,16020,1519845790686/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com%2C16020%2C1519845790686.default.1520193926515
> for DFSClient_NONMAPREDUCE_1077513762_1
> 2018-03-05 04:05:39,761 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: BLOCK*
> blk_1085498930_11758129\{UCState=COMMITTED, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-32c7e479-3845-4a44-adf1-831edec7506b:NORMAL:10.0.0.219:50010|RBW],
>
> ReplicaUC[[DISK]DS-a9a5d653-c049-463d-8e4a-d1f0dc14409c:NORMAL:10.0.0.220:50010|RBW],
>
> ReplicaUC[[DISK]DS-f2b7c04a-b724-4c69-abbf-d2e416f70706:NORMAL:10.0.0.218:50010|RBW]]}
> is not COMPLETE (ucState = COMMITTED, replication# = 0 < minimum = 2) in
> file
> /hbase/WALs/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com,16020,1519845790686/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com%2C16020%2C1519845790686.default.1520193926515
> 2018-03-05 04:05:39,761 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.0.0.220:50010 is added to
> blk_1085498930_11758129\{UCState=COMMITTED, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-32c7e479-3845-4a44-adf1-831edec7506b:NORMAL:10.0.0.219:50010|RBW],
>
> ReplicaUC[[DISK]DS-a9a5d653-c049-463d-8e4a-d1f0dc14409c:NORMAL:10.0.0.220:50010|RBW],
>
> ReplicaUC[[DISK]DS-f2b7c04a-b724-4c69-abbf-d2e416f70706:NORMAL:10.0.0.218:50010|RBW]]}
> size 2054413
> 2018-03-05 04:05:39,761 INFO BlockStateChange: BLOCK
> NameSystem.addToCorruptReplicasMap: blk_1085498930 added as corrupt on
> 10.0.0.219:50010 by
> hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com/10.0.0.219 because block is
> COMMITTED and reported length 2054413 does not match length in block map
> 141232
> 2018-03-05 04:05:39,762 INFO BlockStateChange: BLOCK
> NameSystem.addToCorruptReplicasMap: blk_1085498930 added as
[jira] [Updated] (HDFS-8893) DNs with failed volumes stop serving during rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-8893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-8893:
-
Target Version/s: 3.3.0 (was: 3.2.0)
> DNs with failed volumes stop serving during rolling upgrade
> ---
>
> Key: HDFS-8893
> URL: https://issues.apache.org/jira/browse/HDFS-8893
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Daryn Sharp
>Priority: Critical
>
> When a rolling upgrade starts, all DNs try to write a rolling_upgrade marker
> to each of their volumes. If one of the volumes is bad, this will fail. When
> this failure happens, the DN does not update the key it received from the NN.
> Unfortunately we had one failed volume on all the 3 datanodes which were
> having replica.
> Keys expire after 20 hours so at about 20 hours into the rolling upgrade, the
> DNs with failed volumes will stop serving clients.
> Here is the stack trace on the datanode size:
> {noformat}
> 2015-08-11 07:32:28,827 [DataNode: heartbeating to 8020] WARN
> datanode.DataNode: IOException in offerService
> java.io.IOException: Read-only file system
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:947)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.setRollingUpgradeMarkers(BlockPoolSliceStorage.java:721)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.setRollingUpgradeMarker(DataStorage.java:173)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.setRollingUpgradeMarker(FsDatasetImpl.java:2357)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.signalRollingUpgrade(BPOfferService.java:480)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.handleRollingUpgradeStatus(BPServiceActor.java:626)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:677)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:833)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8893) DNs with failed volumes stop serving during rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-8893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608734#comment-16608734
]
Rushabh S Shah commented on HDFS-8893:
--
Moved to 3.3.0
> DNs with failed volumes stop serving during rolling upgrade
> ---
>
> Key: HDFS-8893
> URL: https://issues.apache.org/jira/browse/HDFS-8893
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Daryn Sharp
>Priority: Critical
>
> When a rolling upgrade starts, all DNs try to write a rolling_upgrade marker
> to each of their volumes. If one of the volumes is bad, this will fail. When
> this failure happens, the DN does not update the key it received from the NN.
> Unfortunately we had one failed volume on all the 3 datanodes which were
> having replica.
> Keys expire after 20 hours so at about 20 hours into the rolling upgrade, the
> DNs with failed volumes will stop serving clients.
> Here is the stack trace on the datanode size:
> {noformat}
> 2015-08-11 07:32:28,827 [DataNode: heartbeating to 8020] WARN
> datanode.DataNode: IOException in offerService
> java.io.IOException: Read-only file system
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:947)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.setRollingUpgradeMarkers(BlockPoolSliceStorage.java:721)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.setRollingUpgradeMarker(DataStorage.java:173)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.setRollingUpgradeMarker(FsDatasetImpl.java:2357)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.signalRollingUpgrade(BPOfferService.java:480)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.handleRollingUpgradeStatus(BPServiceActor.java:626)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:677)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:833)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13806) EC: No error message for unsetting EC policy of the directory inherits the erasure coding policy from an ancestor directory
[ https://issues.apache.org/jira/browse/HDFS-13806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608730#comment-16608730 ] Hudson commented on HDFS-13806: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14910 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14910/]) HDFS-13806. EC: No error message for unsetting EC policy of the (vinayakumarb: rev 30eceec3420fc6be00d3878ba787bd9518d3ca0e) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/ECAdmin.java * (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/NoECPolicySetException.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirErasureCodingOp.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestUnsetAndChangeDirectoryEcPolicy.java > EC: No error message for unsetting EC policy of the directory inherits the > erasure coding policy from an ancestor directory > --- > > Key: HDFS-13806 > URL: https://issues.apache.org/jira/browse/HDFS-13806 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 > Environment: 3 Node SUSE Linux cluster >Reporter: Souryakanta Dwivedy >Assignee: Ayush Saxena >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-13806-01.patch, HDFS-13806-02.patch, > HDFS-13806-03.patch, HDFS-13806-04.patch, HDFS-13806-05.patch, > HDFS-13806-06.patch, No_error_unset_ec_policy.png > > > No error message thrown for unsetting EC policy of the directory inherits the > erasure coding policy from an ancestor directory > Steps :- > -- > * Create a Directory > - Set EC policy for the Directory > - Create a file in-side that Directory > - Create a sub-directory inside the parent directory > - Check both the file and sub-directory inherit the EC policy from parent > directory > - Try to unset EC Policy for the file and check it will throw error as [ > Cannot unset an erasure coding policy on a file] > - Try to unset EC Policy for the sub-directory and check it will throw a > success message as [Unset erasure coding policy from ] > instead of throwing the error message,which is wrong behavior > Actual output :- > No proper error message thrown for unsetting EC policy of the directory > inherits the erasure coding policy from an ancestor directory > A success message is displayed instead of throwing an error message > Expected output :- > > Proper error message should be thrown while trying to unset EC policy of the > directory inherits the erasure coding policy from an ancestor directory > like error message thrown while unsetting the EC policy of a file inherits > the erasure coding policy from an ancestor directory -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13895) EC: Fix Intermittent Failure in TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
[
https://issues.apache.org/jira/browse/HDFS-13895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608731#comment-16608731
]
Hudson commented on HDFS-13895:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14910 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14910/])
HDFS-13895. EC: Fix Intermittent Failure in (vinayakumarb: rev
eef3bafae8bc0fd31506637cd131599ec97f362f)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java
> EC: Fix Intermittent Failure in
> TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
> ---
>
> Key: HDFS-13895
> URL: https://issues.apache.org/jira/browse/HDFS-13895
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Fix For: 3.2.0, 3.1.2
>
> Attachments: HDFS-13895.patch
>
>
> https://builds.apache.org/job/PreCommit-HDFS-Build/24893/testReport/org.apache.hadoop.hdfs/TestDFSStripedOutputStreamWithFailureWithRandomECPolicy/testCloseWithExceptionsInStreamer/
> {noformat}
> java.io.IOException: Failed: the number of failed blocks = 2 > the number of
> parity blocks = 1
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.checkStreamers(DFSStripedOutputStream.java:395)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.checkStreamerFailures(DFSStripedOutputStream.java:623)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.writeChunk(DFSStripedOutputStream.java:566)
> at
> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:217)
> at
> org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:164)
> at
> org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:145)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.closeImpl(DFSStripedOutputStream.java:1166)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13744) OIV tool should better handle control characters present in file or directory names
[
https://issues.apache.org/jira/browse/HDFS-13744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608719#comment-16608719
]
Sunil Govindan commented on HDFS-13744:
---
Hi [~mackrorysd] Looks like patch is committed, but issue is not closed. Could
you please close this if its fine. Thank you.
> OIV tool should better handle control characters present in file or directory
> names
> ---
>
> Key: HDFS-13744
> URL: https://issues.apache.org/jira/browse/HDFS-13744
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, tools
>Affects Versions: 2.6.5, 2.9.1, 2.8.4, 2.7.6, 3.0.3
>Reporter: Zsolt Venczel
>Assignee: Zsolt Venczel
>Priority: Critical
> Attachments: HDFS-13744.01.patch, HDFS-13744.02.patch,
> HDFS-13744.03.patch
>
>
> In certain cases when control characters or white space is present in file or
> directory names OIV tool processors can export data in a misleading format.
> In the below examples we have EXAMPLE_NAME as a file and a directory name
> where the directory has a line feed character at the end (the actual
> production case has multiple line feeds and multiple spaces)
> * Delimited processor case:
> ** misleading example:
> {code:java}
> /user/data/EXAMPLE_NAME
> ,0,2017-04-24 04:34,1969-12-31 16:00,0,0,0,-1,-1,drwxrwxr-x+,user,group
> /user/data/EXAMPLE_NAME,2016-08-26 03:00,2017-05-16
> 10:05,134217728,1,520,0,0,-rw-rwxr--+,user,group
> {code}
> *
> ** expected example as suggested by
> [https://tools.ietf.org/html/rfc4180#section-2]:
> {code:java}
> "/user/data/EXAMPLE_NAME%x0A",0,2017-04-24 04:34,1969-12-31
> 16:00,0,0,0,-1,-1,drwxrwxr-x+,user,group
> "/user/data/EXAMPLE_NAME",2016-08-26 03:00,2017-05-16
> 10:05,134217728,1,520,0,0,-rw-rwxr--+,user,group
> {code}
> * XML processor case:
> ** misleading example:
> {code:java}
> 479867791DIRECTORYEXAMPLE_NAME
> 1493033668294user:group:0775
> 113632535FILEEXAMPLE_NAME314722056575041494954320141134217728user:group:0674
> {code}
> *
> ** expected example as specified in
> [https://www.w3.org/TR/REC-xml/#sec-line-ends]:
> {code:java}
> 479867791DIRECTORYEXAMPLE_NAME#xA1493033668294user:group:0775
> 113632535FILEEXAMPLE_NAME314722056575041494954320141134217728user:group:0674
> {code}
> * JSON:
> The OIV Web Processor behaves correctly and produces the following:
> {code:java}
> {
> "FileStatuses": {
> "FileStatus": [
> {
> "fileId": 113632535,
> "accessTime": 1494954320141,
> "replication": 3,
> "owner": "user",
> "length": 520,
> "permission": "674",
> "blockSize": 134217728,
> "modificationTime": 1472205657504,
> "type": "FILE",
> "group": "group",
> "childrenNum": 0,
> "pathSuffix": "EXAMPLE_NAME"
> },
> {
> "fileId": 479867791,
> "accessTime": 0,
> "replication": 0,
> "owner": "user",
> "length": 0,
> "permission": "775",
> "blockSize": 0,
> "modificationTime": 1493033668294,
> "type": "DIRECTORY",
> "group": "group",
> "childrenNum": 0,
> "pathSuffix": "EXAMPLE_NAME\n"
> }
> ]
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13243) Get CorruptBlock because of calling close and sync in same time
[
https://issues.apache.org/jira/browse/HDFS-13243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608710#comment-16608710
]
Sunil Govindan commented on HDFS-13243:
---
Ping again : [~gzh1992n]
> Get CorruptBlock because of calling close and sync in same time
> ---
>
> Key: HDFS-13243
> URL: https://issues.apache.org/jira/browse/HDFS-13243
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2, 3.2.0
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>Priority: Critical
> Attachments: HDFS-13243-v1.patch, HDFS-13243-v2.patch,
> HDFS-13243-v3.patch, HDFS-13243-v4.patch, HDFS-13243-v5.patch,
> HDFS-13243-v6.patch
>
>
> HDFS File might get broken because of corrupt block(s) that could be produced
> by calling close and sync in the same time.
> When calling close was not successful, UCBlock status would change to
> COMMITTED, and if a sync request gets popped from queue and processed, sync
> operation would change the last block length.
> After that, DataNode would report all received block to NameNode, and will
> check Block length of all COMMITTED Blocks. But the block length was already
> different between recorded in NameNode memory and reported by DataNode, and
> consequently, the last block is marked as corruptted because of inconsistent
> length.
>
> {panel:title=Log in my hdfs}
> 2018-03-05 04:05:39,261 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> allocate blk_1085498930_11758129\{UCState=UNDER_CONSTRUCTION,
> truncateBlock=null, primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-32c7e479-3845-4a44-adf1-831edec7506b:NORMAL:10.0.0.219:50010|RBW],
>
> ReplicaUC[[DISK]DS-a9a5d653-c049-463d-8e4a-d1f0dc14409c:NORMAL:10.0.0.220:50010|RBW],
>
> ReplicaUC[[DISK]DS-f2b7c04a-b724-4c69-abbf-d2e416f70706:NORMAL:10.0.0.218:50010|RBW]]}
> for
> /hbase/WALs/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com,16020,1519845790686/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com%2C16020%2C1519845790686.default.1520193926515
> 2018-03-05 04:05:39,760 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> fsync:
> /hbase/WALs/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com,16020,1519845790686/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com%2C16020%2C1519845790686.default.1520193926515
> for DFSClient_NONMAPREDUCE_1077513762_1
> 2018-03-05 04:05:39,761 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: BLOCK*
> blk_1085498930_11758129\{UCState=COMMITTED, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-32c7e479-3845-4a44-adf1-831edec7506b:NORMAL:10.0.0.219:50010|RBW],
>
> ReplicaUC[[DISK]DS-a9a5d653-c049-463d-8e4a-d1f0dc14409c:NORMAL:10.0.0.220:50010|RBW],
>
> ReplicaUC[[DISK]DS-f2b7c04a-b724-4c69-abbf-d2e416f70706:NORMAL:10.0.0.218:50010|RBW]]}
> is not COMPLETE (ucState = COMMITTED, replication# = 0 < minimum = 2) in
> file
> /hbase/WALs/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com,16020,1519845790686/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com%2C16020%2C1519845790686.default.1520193926515
> 2018-03-05 04:05:39,761 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.0.0.220:50010 is added to
> blk_1085498930_11758129\{UCState=COMMITTED, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-32c7e479-3845-4a44-adf1-831edec7506b:NORMAL:10.0.0.219:50010|RBW],
>
> ReplicaUC[[DISK]DS-a9a5d653-c049-463d-8e4a-d1f0dc14409c:NORMAL:10.0.0.220:50010|RBW],
>
> ReplicaUC[[DISK]DS-f2b7c04a-b724-4c69-abbf-d2e416f70706:NORMAL:10.0.0.218:50010|RBW]]}
> size 2054413
> 2018-03-05 04:05:39,761 INFO BlockStateChange: BLOCK
> NameSystem.addToCorruptReplicasMap: blk_1085498930 added as corrupt on
> 10.0.0.219:50010 by
> hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com/10.0.0.219 because block is
> COMMITTED and reported length 2054413 does not match length in block map
> 141232
> 2018-03-05 04:05:39,762 INFO BlockStateChange: BLOCK
> NameSystem.addToCorruptReplicasMap: blk_1085498930 added as corrupt on
> 10.0.0.218:50010 by
> hb-j5e517al6xib80rkb-004.hbase.rds.aliyuncs.com/10.0.0.218 because block is
> COMMITTED and reported length 2054413 does not match length in block map
> 141232
> 2018-03-05 04:05:40,162 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: BLOCK*
> blk_1085498930_11758129\{UCState=COMMITTED, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-32c7e479-3845-4a44-adf1-831edec7506b:NORMAL:10.0.0.219:50010|RBW],
>
> ReplicaUC[[DISK]DS-a9a5d653-c049-463d-8e4a-d1f0dc14409c:NORMAL:10.0.0.220:50010|RBW],
>
> ReplicaUC[[DISK]DS-f2b7c04a-b724-4c69-abbf-d2e416f70706:NORMAL:10.0.0.218:50010|RBW]]}
> is not COMPLETE (ucState = COMMITTED, replication# = 3 >= minimum = 2) in
> file
>
[jira] [Commented] (HDFS-12452) TestDataNodeVolumeFailureReporting fails in trunk Jenkins runs
[
https://issues.apache.org/jira/browse/HDFS-12452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608709#comment-16608709
]
Sunil Govindan commented on HDFS-12452:
---
Ping again: [~xyao]
> TestDataNodeVolumeFailureReporting fails in trunk Jenkins runs
> --
>
> Key: HDFS-12452
> URL: https://issues.apache.org/jira/browse/HDFS-12452
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Arpit Agarwal
>Assignee: Xiaoyu Yao
>Priority: Critical
> Labels: flaky-test
> Attachments: HDFS-12452.001.patch, HDFS-12452.002.patch
>
>
> TestDataNodeVolumeFailureReporting#testSuccessiveVolumeFailures fails
> frequently in Jenkins runs but it passes locally on my dev machine.
> e.g.
> https://builds.apache.org/job/PreCommit-HDFS-Build/21134/testReport/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailureReporting/testSuccessiveVolumeFailures/
> {code}
> Error Message
> test timed out after 12 milliseconds
> Stacktrace
> java.lang.Exception: test timed out after 12 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.waitReplication(DFSTestUtil.java:761)
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting.testSuccessiveVolumeFailures(TestDataNodeVolumeFailureReporting.java:189)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8893) DNs with failed volumes stop serving during rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-8893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608707#comment-16608707
]
Sunil Govindan commented on HDFS-8893:
--
ping [~daryn] [~shahrs87]
> DNs with failed volumes stop serving during rolling upgrade
> ---
>
> Key: HDFS-8893
> URL: https://issues.apache.org/jira/browse/HDFS-8893
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Daryn Sharp
>Priority: Critical
>
> When a rolling upgrade starts, all DNs try to write a rolling_upgrade marker
> to each of their volumes. If one of the volumes is bad, this will fail. When
> this failure happens, the DN does not update the key it received from the NN.
> Unfortunately we had one failed volume on all the 3 datanodes which were
> having replica.
> Keys expire after 20 hours so at about 20 hours into the rolling upgrade, the
> DNs with failed volumes will stop serving clients.
> Here is the stack trace on the datanode size:
> {noformat}
> 2015-08-11 07:32:28,827 [DataNode: heartbeating to 8020] WARN
> datanode.DataNode: IOException in offerService
> java.io.IOException: Read-only file system
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:947)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.setRollingUpgradeMarkers(BlockPoolSliceStorage.java:721)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.setRollingUpgradeMarker(DataStorage.java:173)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.setRollingUpgradeMarker(FsDatasetImpl.java:2357)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.signalRollingUpgrade(BPOfferService.java:480)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.handleRollingUpgradeStatus(BPServiceActor.java:626)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:677)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:833)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12049) Recommissioning live nodes stalls the NN
[ https://issues.apache.org/jira/browse/HDFS-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608706#comment-16608706 ] Sunil Govindan commented on HDFS-12049: --- Hi [~daryn], Could u please help to check on this issue. As there is no progress and code freeze for 3.2.0 is nearing, we can move to 3.3.0 if there are no immediate plans. > Recommissioning live nodes stalls the NN > > > Key: HDFS-12049 > URL: https://issues.apache.org/jira/browse/HDFS-12049 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Daryn Sharp >Priority: Critical > > A node refresh will recommission included nodes that are alive and in > decommissioning or decommissioned state. The recommission will scan all > blocks on the node, find over replicated blocks, chose an excess, queue an > invalidate. > The process is expensive and worsened by overhead of storage types (even when > not in use). It can be especially devastating because the write lock is held > for the entire node refresh. _Recommissioning 67 nodes with ~500k > blocks/node stalled rpc services for over 4 mins._ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11310) Reduce the performance impact of the balancer (trunk port)
[ https://issues.apache.org/jira/browse/HDFS-11310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608702#comment-16608702 ] Sunil Govindan commented on HDFS-11310: --- Thanks [~daryn]. 3.2.0 code freeze is nearby (15th Sept), could u please share the plan for this Jira or we may need to move it out. > Reduce the performance impact of the balancer (trunk port) > -- > > Key: HDFS-11310 > URL: https://issues.apache.org/jira/browse/HDFS-11310 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Daryn Sharp >Priority: Critical > > HDFS-7967 introduced a highly performant balancer getBlocks() query that > scales to large/dense clusters. The simple design implementation depends on > the triplets data structure. HDFS-9260 removed the triplets which > fundamentally changes the implementation. Either that patch must be reverted > or the getBlocks() patch needs reimplementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13596) NN restart fails after RollingUpgrade from 2.x to 3.x
[
https://issues.apache.org/jira/browse/HDFS-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608693#comment-16608693
]
Sunil Govindan commented on HDFS-13596:
---
Code freeze for 3.2.0 is nearing (15th Sept) and there are no contributors for
this yet. Since this is a blocker, pinging [~zvenczel] [~hanishakoneru]
[~rajeshhadoop] [~leftnoteasy] [~rohithsharma] [~vinayrpet] [~rakeshr]
[~umamaheswararao] for further steps.
If we wont be able to finish this, I think we will need to move this to next
version.
> NN restart fails after RollingUpgrade from 2.x to 3.x
> -
>
> Key: HDFS-13596
> URL: https://issues.apache.org/jira/browse/HDFS-13596
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Hanisha Koneru
>Assignee: Zsolt Venczel
>Priority: Blocker
>
> After rollingUpgrade NN from 2.x and 3.x, if the NN is restarted, it fails
> while replaying edit logs.
> * After NN is started with rollingUpgrade, the layoutVersion written to
> editLogs (before finalizing the upgrade) is the pre-upgrade layout version
> (so as to support downgrade).
> * When writing transactions to log, NN writes as per the current layout
> version. In 3.x, erasureCoding bits are added to the editLog transactions.
> * So any edit log written after the upgrade and before finalizing the
> upgrade will have the old layout version but the new format of transactions.
> * When NN is restarted and the edit logs are replayed, the NN reads the old
> layout version from the editLog file. When parsing the transactions, it
> assumes that the transactions are also from the previous layout and hence
> skips parsing the erasureCoding bits.
> * This cascades into reading the wrong set of bits for other fields and
> leads to NN shutting down.
> Sample error output:
> {code:java}
> java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected
> length 16
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:74)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:86)
> at
> org.apache.hadoop.ipc.RetryCache$CacheEntryWithPayload.(RetryCache.java:163)
> at
> org.apache.hadoop.ipc.RetryCache.addCacheEntryWithPayload(RetryCache.java:322)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheEntryWithPayload(FSNamesystem.java:960)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:397)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:249)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:888)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1086)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:937)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:910)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710)
> 2018-05-17 19:10:06,522 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: java.lang.IllegalStateException: Cannot skip to less
> than the current value (=16389), where newValue=16388
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.resetLastInodeId(FSDirectory.java:1945)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:298)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:888)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1086)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714)
> at
>
[jira] [Updated] (HDFS-13895) EC: Fix Intermittent Failure in TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
[
https://issues.apache.org/jira/browse/HDFS-13895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-13895:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.2
3.2.0
Status: Resolved (was: Patch Available)
+1
Committed to trunk and branch-3.1
Thanks [~ayushtkn]
> EC: Fix Intermittent Failure in
> TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
> ---
>
> Key: HDFS-13895
> URL: https://issues.apache.org/jira/browse/HDFS-13895
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Fix For: 3.2.0, 3.1.2
>
> Attachments: HDFS-13895.patch
>
>
> https://builds.apache.org/job/PreCommit-HDFS-Build/24893/testReport/org.apache.hadoop.hdfs/TestDFSStripedOutputStreamWithFailureWithRandomECPolicy/testCloseWithExceptionsInStreamer/
> {noformat}
> java.io.IOException: Failed: the number of failed blocks = 2 > the number of
> parity blocks = 1
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.checkStreamers(DFSStripedOutputStream.java:395)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.checkStreamerFailures(DFSStripedOutputStream.java:623)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.writeChunk(DFSStripedOutputStream.java:566)
> at
> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:217)
> at
> org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:164)
> at
> org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:145)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.closeImpl(DFSStripedOutputStream.java:1166)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13806) EC: No error message for unsetting EC policy of the directory inherits the erasure coding policy from an ancestor directory
[ https://issues.apache.org/jira/browse/HDFS-13806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-13806: - Resolution: Fixed Hadoop Flags: Incompatible change,Reviewed Fix Version/s: 3.2.0 Release Note: After this change, attempt to unsetErasureCodingPolicy() on a directory without EC policy explicitly set on it, will get NoECPolicySetException. Status: Resolved (was: Patch Available) +1 Committed to trunk only as this will be incompatible. Thanks [~ayushtkn] for contribution. Thanks [~xiaochen] and [~Sammi] for reviews > EC: No error message for unsetting EC policy of the directory inherits the > erasure coding policy from an ancestor directory > --- > > Key: HDFS-13806 > URL: https://issues.apache.org/jira/browse/HDFS-13806 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 > Environment: 3 Node SUSE Linux cluster >Reporter: Souryakanta Dwivedy >Assignee: Ayush Saxena >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-13806-01.patch, HDFS-13806-02.patch, > HDFS-13806-03.patch, HDFS-13806-04.patch, HDFS-13806-05.patch, > HDFS-13806-06.patch, No_error_unset_ec_policy.png > > > No error message thrown for unsetting EC policy of the directory inherits the > erasure coding policy from an ancestor directory > Steps :- > -- > * Create a Directory > - Set EC policy for the Directory > - Create a file in-side that Directory > - Create a sub-directory inside the parent directory > - Check both the file and sub-directory inherit the EC policy from parent > directory > - Try to unset EC Policy for the file and check it will throw error as [ > Cannot unset an erasure coding policy on a file] > - Try to unset EC Policy for the sub-directory and check it will throw a > success message as [Unset erasure coding policy from ] > instead of throwing the error message,which is wrong behavior > Actual output :- > No proper error message thrown for unsetting EC policy of the directory > inherits the erasure coding policy from an ancestor directory > A success message is displayed instead of throwing an error message > Expected output :- > > Proper error message should be thrown while trying to unset EC policy of the > directory inherits the erasure coding policy from an ancestor directory > like error message thrown while unsetting the EC policy of a file inherits > the erasure coding policy from an ancestor directory -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13445) Hostname is not getting resolved when we pass IP in webhdfs request.
[ https://issues.apache.org/jira/browse/HDFS-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ranith Sardar updated HDFS-13445: - Status: Patch Available (was: Open) > Hostname is not getting resolved when we pass IP in webhdfs request. > > > Key: HDFS-13445 > URL: https://issues.apache.org/jira/browse/HDFS-13445 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, security, webhdfs >Affects Versions: 2.8.3 >Reporter: Ranith Sardar >Assignee: Ranith Sardar >Priority: Major > Attachments: HDFS-13445.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13866) Reset exitcode to -1 when invalid params are inputed for dfsrouteradmin commands
[ https://issues.apache.org/jira/browse/HDFS-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ranith Sardar reassigned HDFS-13866: Assignee: Ranith Sardar > Reset exitcode to -1 when invalid params are inputed for dfsrouteradmin > commands > > > Key: HDFS-13866 > URL: https://issues.apache.org/jira/browse/HDFS-13866 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation >Affects Versions: 3.0.0 >Reporter: Ranith Sardar >Assignee: Ranith Sardar >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13869) RBF: Handle NPE for NamenodeBeanMetrics#getFederationMetrics()
[
https://issues.apache.org/jira/browse/HDFS-13869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608629#comment-16608629
]
Ranith Sardar edited comment on HDFS-13869 at 9/10/18 2:11 AM:
---
We will get this exception when the metrics are not initialized , So used
{{TestDisableRouterQuota class.}}
Yes, Both the cases are different for "Metrics is not initialized" exception.
But just handled the NPE scenario.
In my next patch, i will update correct the exception msg and will write new
class for UT.
was (Author: ranith):
We will get this exception when the metrics are not initialized , So used
{{TestDisableRouterQuota class.}}
Yes, Both the cases are different for "Metrics is not initialized" exception.
But just handled the NPE scenario.
> RBF: Handle NPE for NamenodeBeanMetrics#getFederationMetrics()
> --
>
> Key: HDFS-13869
> URL: https://issues.apache.org/jira/browse/HDFS-13869
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0
>Reporter: Surendra Singh Lilhore
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-13869-002.diff, HDFS-13869-003.diff,
> HDFS-13869-004.patch, HDFS-13869.patch
>
>
> {code:java}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics.getUsed(NamenodeBeanMetrics.java:205)
> at
> org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics.getCapacityUsed(NamenodeBeanMetrics.java:519)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
> ngMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13869) RBF: Handle NPE for NamenodeBeanMetrics#getFederationMetrics()
[
https://issues.apache.org/jira/browse/HDFS-13869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608629#comment-16608629
]
Ranith Sardar commented on HDFS-13869:
--
We will get this exception when the metrics are not initialized , So used
{{TestDisableRouterQuota class.}}
Yes, Both the cases are different for "Metrics is not initialized" exception.
But just handled the NPE scenario.
> RBF: Handle NPE for NamenodeBeanMetrics#getFederationMetrics()
> --
>
> Key: HDFS-13869
> URL: https://issues.apache.org/jira/browse/HDFS-13869
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0
>Reporter: Surendra Singh Lilhore
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-13869-002.diff, HDFS-13869-003.diff,
> HDFS-13869-004.patch, HDFS-13869.patch
>
>
> {code:java}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics.getUsed(NamenodeBeanMetrics.java:205)
> at
> org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics.getCapacityUsed(NamenodeBeanMetrics.java:519)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
> ngMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12716) 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available
[ https://issues.apache.org/jira/browse/HDFS-12716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608627#comment-16608627 ] Ranith Sardar commented on HDFS-12716: -- Thank you [~brahmareddy]. :) > 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes > to be available > - > > Key: HDFS-12716 > URL: https://issues.apache.org/jira/browse/HDFS-12716 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: usharani >Assignee: Ranith Sardar >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2 > > Attachments: HDFS-12716-branch-2.patch, HDFS-12716.002.patch, > HDFS-12716.003.patch, HDFS-12716.004.patch, HDFS-12716.005.patch, > HDFS-12716.006.patch, HDFS-12716.patch, HDFS-12716_branch-2.patch > > > Currently 'dfs.datanode.failed.volumes.tolerated' supports number of > tolerated failed volumes to be mentioned. This configuration change requires > restart of datanode. Since datanode volumes can be changed dynamically, > keeping this configuration same for all may not be good idea. > Support 'dfs.datanode.failed.volumes.tolerated' to accept special > 'negative value 'x' to tolerate failures of upto "n-x" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-419) ChunkInputStream bulk read api does not read from all the chunks
[ https://issues.apache.org/jira/browse/HDDS-419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-419: --- Status: Patch Available (was: Open) > ChunkInputStream bulk read api does not read from all the chunks > > > Key: HDDS-419 > URL: https://issues.apache.org/jira/browse/HDDS-419 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.2.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Blocker > Fix For: 0.2.1 > > Attachments: HDDS-419.001.patch > > > After enabling of bulk reads with HDDS-408, testDataValidate started failing > because the bulk read api does not read all the chunks from the block. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12716) 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available
[ https://issues.apache.org/jira/browse/HDFS-12716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12716: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.10.0 Status: Resolved (was: Patch Available) Committed to branch-2.. Jenkins precommit is having issue with branch-2,see HADOOP-13951 for details. I verified the fix locally..there was one whitespace, fixed while committing. [~RANith] thanks for contribution. > 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes > to be available > - > > Key: HDFS-12716 > URL: https://issues.apache.org/jira/browse/HDFS-12716 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: usharani >Assignee: Ranith Sardar >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2 > > Attachments: HDFS-12716-branch-2.patch, HDFS-12716.002.patch, > HDFS-12716.003.patch, HDFS-12716.004.patch, HDFS-12716.005.patch, > HDFS-12716.006.patch, HDFS-12716.patch, HDFS-12716_branch-2.patch > > > Currently 'dfs.datanode.failed.volumes.tolerated' supports number of > tolerated failed volumes to be mentioned. This configuration change requires > restart of datanode. Since datanode volumes can be changed dynamically, > keeping this configuration same for all may not be good idea. > Support 'dfs.datanode.failed.volumes.tolerated' to accept special > 'negative value 'x' to tolerate failures of upto "n-x" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-362) Modify functions impacted by SCM chill mode in ScmBlockLocationProtocol
[ https://issues.apache.org/jira/browse/HDDS-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-362: -- Fix Version/s: 0.2.1 > Modify functions impacted by SCM chill mode in ScmBlockLocationProtocol > --- > > Key: HDDS-362 > URL: https://issues.apache.org/jira/browse/HDDS-362 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-362.00.patch, HDDS-362.01.patch > > > Modify functions impacted by SCM chill mode in ScmBlockLocationProtocol -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-421) Resilient DNS resolution in datanode-service
[
https://issues.apache.org/jira/browse/HDDS-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608581#comment-16608581
]
Hadoop QA commented on HDDS-421:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} ozone-0.2.1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
29s{color} | {color:green} ozone-0.2.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} ozone-0.2.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} ozone-0.2.1 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} ozone-0.2.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} ozone-0.2.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} ozone-0.2.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
50s{color} | {color:green} container-service in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDDS-421 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939004/HDDS-421-ozone-0.2.1.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux d442c8bb24d7 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | ozone-0.2.1 / be1ec00 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/1015/testReport/ |
| Max. process+thread count | 407 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/container-service U: hadoop-hdds/container-service |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/1015/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
>
[jira] [Commented] (HDDS-421) Resilient DNS resolution in datanode-service
[ https://issues.apache.org/jira/browse/HDDS-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608575#comment-16608575 ] Elek, Marton commented on HDDS-421: --- Tested with kubernetes. All of the datanodes could be started with this patch. > Resilient DNS resolution in datanode-service > - > > Key: HDDS-421 > URL: https://issues.apache.org/jira/browse/HDDS-421 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-421-ozone-0.2.1.001.patch > > > When I start big clusters on kubernetes I got a very typical error: > If the DNS of the scm is not yet available during the bootup of the datanode: > the datanode won't connect to the scm. It tries to reconnect but the dns > resolution is not repeated. > The problem is in the InitDatanodeState.call(). It calls the getSCMAddresses > which creates the InetSocketAddress-es with using the hadoop utilities. > During the creation of the InetSocketAddress the hadoop utilities try to > resolve the address and save the result to the InetSocketAddress. > The address could be unresolved, but the InitDatanodeState.call will start to > use it (connectionManager.addSCMServer) and there won't be any attempt to > resolve it later. > My small proposal is to return immediately of any of the scm addresses is > unresolved and the main loop of the DatanodeStateMachine will try it again > (together with the DNS resolution part). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-421) Resilient DNS resolution in datanode-service
[ https://issues.apache.org/jira/browse/HDDS-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-421: -- Status: Patch Available (was: Open) > Resilient DNS resolution in datanode-service > - > > Key: HDDS-421 > URL: https://issues.apache.org/jira/browse/HDDS-421 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-421-ozone-0.2.1.001.patch > > > When I start big clusters on kubernetes I got a very typical error: > If the DNS of the scm is not yet available during the bootup of the datanode: > the datanode won't connect to the scm. It tries to reconnect but the dns > resolution is not repeated. > The problem is in the InitDatanodeState.call(). It calls the getSCMAddresses > which creates the InetSocketAddress-es with using the hadoop utilities. > During the creation of the InetSocketAddress the hadoop utilities try to > resolve the address and save the result to the InetSocketAddress. > The address could be unresolved, but the InitDatanodeState.call will start to > use it (connectionManager.addSCMServer) and there won't be any attempt to > resolve it later. > My small proposal is to return immediately of any of the scm addresses is > unresolved and the main loop of the DatanodeStateMachine will try it again > (together with the DNS resolution part). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-421) Resilient DNS resolution in datanode-service
[ https://issues.apache.org/jira/browse/HDDS-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-421: -- Attachment: HDDS-421-ozone-0.2.1.001.patch > Resilient DNS resolution in datanode-service > - > > Key: HDDS-421 > URL: https://issues.apache.org/jira/browse/HDDS-421 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-421-ozone-0.2.1.001.patch > > > When I start big clusters on kubernetes I got a very typical error: > If the DNS of the scm is not yet available during the bootup of the datanode: > the datanode won't connect to the scm. It tries to reconnect but the dns > resolution is not repeated. > The problem is in the InitDatanodeState.call(). It calls the getSCMAddresses > which creates the InetSocketAddress-es with using the hadoop utilities. > During the creation of the InetSocketAddress the hadoop utilities try to > resolve the address and save the result to the InetSocketAddress. > The address could be unresolved, but the InitDatanodeState.call will start to > use it (connectionManager.addSCMServer) and there won't be any attempt to > resolve it later. > My small proposal is to return immediately of any of the scm addresses is > unresolved and the main loop of the DatanodeStateMachine will try it again > (together with the DNS resolution part). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-421) Resilient DNS resolution in datanode-service
Elek, Marton created HDDS-421: - Summary: Resilient DNS resolution in datanode-service Key: HDDS-421 URL: https://issues.apache.org/jira/browse/HDDS-421 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode Reporter: Elek, Marton Assignee: Elek, Marton Fix For: 0.2.1 When I start big clusters on kubernetes I got a very typical error: If the DNS of the scm is not yet available during the bootup of the datanode: the datanode won't connect to the scm. It tries to reconnect but the dns resolution is not repeated. The problem is in the InitDatanodeState.call(). It calls the getSCMAddresses which creates the InetSocketAddress-es with using the hadoop utilities. During the creation of the InetSocketAddress the hadoop utilities try to resolve the address and save the result to the InetSocketAddress. The address could be unresolved, but the InitDatanodeState.call will start to use it (connectionManager.addSCMServer) and there won't be any attempt to resolve it later. My small proposal is to return immediately of any of the scm addresses is unresolved and the main loop of the DatanodeStateMachine will try it again (together with the DNS resolution part). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13245) RBF: State store DBMS implementation
[
https://issues.apache.org/jira/browse/HDFS-13245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608532#comment-16608532
]
Hadoop QA commented on HDFS-13245:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-assemblies {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-assemblies {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
23s{color} | {color:green} hadoop-assemblies in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
34s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}112m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDFS-13245 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-13862) RBF: Router logs are not capturing few of the dfsrouteradmin commands
[ https://issues.apache.org/jira/browse/HDFS-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608529#comment-16608529 ] Hudson commented on HDFS-13862: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14909 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14909/]) HDFS-13862. RBF: Router logs are not capturing few of the dfsrouteradmin (brahma: rev 8944407d938bb8e38c7182a68b4b64d1a7c778ca) * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAdminServer.java > RBF: Router logs are not capturing few of the dfsrouteradmin commands > - > > Key: HDFS-13862 > URL: https://issues.apache.org/jira/browse/HDFS-13862 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Soumyapn >Assignee: Ayush Saxena >Priority: Major > Labels: RBF > Fix For: 3.2.0, 3.1.2 > > Attachments: HDFS-13862-01.patch, HDFS-13862-02.patch, > HDFS-13862-03.patch, HDFS-13862-04.patch, HDFS-13862-05.patch > > > Test Steps : > Below commands are not getting captured in the Router logs. > # Destination entry name in the add command. Log says "Added new mount point > /apps9 to resolver". > # Safemode enter|leave|get commands > # nameservice enable -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13902) Add JMX, conf and stacks menus to the datanode page
[ https://issues.apache.org/jira/browse/HDFS-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608526#comment-16608526 ] Brahma Reddy Battula commented on HDFS-13902: - Journalnode improvement can be handle in separate jira.[~elgoiri],if you agree,I can gohead with commit. > Add JMX, conf and stacks menus to the datanode page > > > Key: HDFS-13902 > URL: https://issues.apache.org/jira/browse/HDFS-13902 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.0.3 >Reporter: fengchuang >Assignee: fengchuang >Priority: Minor > Attachments: HDFS-13902.001.patch > > > Add JMX, conf and stacks menus to the datanode page. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12716) 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available
[ https://issues.apache.org/jira/browse/HDFS-12716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608525#comment-16608525 ] Brahma Reddy Battula commented on HDFS-12716: - Looks some problem branch-2 jenkins. Again I triggered the jenkins. > 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes > to be available > - > > Key: HDFS-12716 > URL: https://issues.apache.org/jira/browse/HDFS-12716 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: usharani >Assignee: Ranith Sardar >Priority: Major > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HDFS-12716-branch-2.patch, HDFS-12716.002.patch, > HDFS-12716.003.patch, HDFS-12716.004.patch, HDFS-12716.005.patch, > HDFS-12716.006.patch, HDFS-12716.patch, HDFS-12716_branch-2.patch > > > Currently 'dfs.datanode.failed.volumes.tolerated' supports number of > tolerated failed volumes to be mentioned. This configuration change requires > restart of datanode. Since datanode volumes can be changed dynamically, > keeping this configuration same for all may not be good idea. > Support 'dfs.datanode.failed.volumes.tolerated' to accept special > 'negative value 'x' to tolerate failures of upto "n-x" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13862) RBF: Router logs are not capturing few of the dfsrouteradmin commands
[ https://issues.apache.org/jira/browse/HDFS-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13862: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.2 3.2.0 Status: Resolved (was: Patch Available) Committed to Trunk and branch-3.1. [~ayushtkn] thanks for contribution,[~SoumyaPN] thanks for reporting and thanks to [~elgoiri] for additional review. > RBF: Router logs are not capturing few of the dfsrouteradmin commands > - > > Key: HDFS-13862 > URL: https://issues.apache.org/jira/browse/HDFS-13862 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Soumyapn >Assignee: Ayush Saxena >Priority: Major > Labels: RBF > Fix For: 3.2.0, 3.1.2 > > Attachments: HDFS-13862-01.patch, HDFS-13862-02.patch, > HDFS-13862-03.patch, HDFS-13862-04.patch, HDFS-13862-05.patch > > > Test Steps : > Below commands are not getting captured in the Router logs. > # Destination entry name in the add command. Log says "Added new mount point > /apps9 to resolver". > # Safemode enter|leave|get commands > # nameservice enable -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13862) RBF: Router logs are not capturing few of the dfsrouteradmin commands
[
https://issues.apache.org/jira/browse/HDFS-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608518#comment-16608518
]
Hadoop QA commented on HDFS-13862:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m
11s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDFS-13862 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12938991/HDFS-13862-05.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6b1c59745faf 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d924ca2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25012/testReport/ |
| Max. process+thread count | 956 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25012/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Router logs are not capturing few of
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608515#comment-16608515 ] Brahma Reddy Battula commented on HDFS-13532: - [~crh] thanks for organising the meeting and detailed design doc.Hope you can update MOM. I am favourable to approach 1(which complete moves token life cycle to Router).As, we need to consider the additional cost also. 1) Please update cons for approach 1. 2) For syncing token across the routers,May be we can refresh/sync thread like HDFS-13443 3) Might need to handle KMS token also.? [~daryn]/[~lmccay]/[~vinayrpet] if you get chance,kindly review the design. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, > RBF-DelegationToken-Approach1b.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13470) RBF: Add Browse the Filesystem button to the UI
[
https://issues.apache.org/jira/browse/HDFS-13470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608513#comment-16608513
]
Hadoop QA commented on HDFS-13470:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
29m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 35 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDFS-13470 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919518/HDFS-13470.000.patch |
| Optional Tests | dupname asflicense shadedclient |
| uname | Linux 19a383054e4f 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d924ca2 |
| maven | version: Apache Maven 3.3.9 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/25010/artifact/out/whitespace-tabs.txt
|
| Max. process+thread count | 335 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25010/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Add Browse the Filesystem button to the UI
> ---
>
> Key: HDFS-13470
> URL: https://issues.apache.org/jira/browse/HDFS-13470
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-13470.000.patch
>
>
> After HDFS-12512 added WebHDFS, we can add the support to browse the
> filesystem to the UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13862) RBF: Router logs are not capturing few of the dfsrouteradmin commands
[ https://issues.apache.org/jira/browse/HDFS-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608502#comment-16608502 ] Brahma Reddy Battula commented on HDFS-13862: - +1,Latest patch.Pending for jenkins. > RBF: Router logs are not capturing few of the dfsrouteradmin commands > - > > Key: HDFS-13862 > URL: https://issues.apache.org/jira/browse/HDFS-13862 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Soumyapn >Assignee: Ayush Saxena >Priority: Major > Labels: RBF > Attachments: HDFS-13862-01.patch, HDFS-13862-02.patch, > HDFS-13862-03.patch, HDFS-13862-04.patch, HDFS-13862-05.patch > > > Test Steps : > Below commands are not getting captured in the Router logs. > # Destination entry name in the add command. Log says "Added new mount point > /apps9 to resolver". > # Safemode enter|leave|get commands > # nameservice enable -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13834) RBF: Connection creator thread should catch Throwable
[
https://issues.apache.org/jira/browse/HDFS-13834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-13834:
Parent Issue: HDFS-13891 (was: HDFS-12615)
> RBF: Connection creator thread should catch Throwable
> -
>
> Key: HDFS-13834
> URL: https://issues.apache.org/jira/browse/HDFS-13834
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Critical
> Attachments: HDFS-13834.0.patch, HDFS-13834.1.patch
>
>
> Connection creator thread is a single thread thats responsible for creating
> all downstream namenode connections.
> This is very critical thread and hence should not die understand
> exception/error scenarios.
> We saw this behavior in production systems where the thread died leaving the
> router process in bad state.
> The thread should also catch a generic error/exception.
> {code}
> @Override
> public void run() {
> while (this.running) {
> try {
> ConnectionPool pool = this.queue.take();
> try {
> int total = pool.getNumConnections();
> int active = pool.getNumActiveConnections();
> if (pool.getNumConnections() < pool.getMaxSize() &&
> active >= MIN_ACTIVE_RATIO * total) {
> ConnectionContext conn = pool.newConnection();
> pool.addConnection(conn);
> } else {
> LOG.debug("Cannot add more than {} connections to {}",
> pool.getMaxSize(), pool);
> }
> } catch (IOException e) {
> LOG.error("Cannot create a new connection", e);
> }
> } catch (InterruptedException e) {
> LOG.error("The connection creator was interrupted");
> this.running = false;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13469) RBF: Support InodeID in the Router
[ https://issues.apache.org/jira/browse/HDFS-13469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13469: Parent Issue: HDFS-13891 (was: HDFS-12615) > RBF: Support InodeID in the Router > -- > > Key: HDFS-13469 > URL: https://issues.apache.org/jira/browse/HDFS-13469 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Priority: Major > > The Namenode supports identifying files through inode identifiers. > Currently the Router does not handle this properly, we need to add this > functionality. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13219) RBF: Cluster information on Router is not correct when the Federation shares datanodes
[
https://issues.apache.org/jira/browse/HDFS-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-13219:
Parent Issue: HDFS-13891 (was: HDFS-12615)
> RBF: Cluster information on Router is not correct when the Federation shares
> datanodes
> --
>
> Key: HDFS-13219
> URL: https://issues.apache.org/jira/browse/HDFS-13219
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.9.0
>Reporter: Tao Jie
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png, screenshot-5.png
>
>
> Now summary information on Router website aggregates summary of each
> nameservice. However in a typical federation cluster deployment, datanodes
> are shared among nameservices. Consider we have 2 namespaces and 100
> datanodes in one cluster. 100 datanodes are available for each namespace, but
> we see 200 datanodes on the router website. So does other information such as
> {{Total capacity}}, {{Remaining capacity}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13270) RBF: Router audit logger
[ https://issues.apache.org/jira/browse/HDFS-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13270: Parent Issue: HDFS-13891 (was: HDFS-12615) > RBF: Router audit logger > > > Key: HDFS-13270 > URL: https://issues.apache.org/jira/browse/HDFS-13270 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Affects Versions: 3.2.0 >Reporter: maobaolong >Priority: Major > > We can use router auditlogger to log the client info and cmd, because the > FSNamesystem#Auditlogger's log think the client are all from router. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13254) RBF: Cannot mv/cp file cross namespace
[ https://issues.apache.org/jira/browse/HDFS-13254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13254: Parent Issue: HDFS-13891 (was: HDFS-12615) > RBF: Cannot mv/cp file cross namespace > -- > > Key: HDFS-13254 > URL: https://issues.apache.org/jira/browse/HDFS-13254 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Weiwei Wu >Priority: Major > > When I try to mv a file from a namespace to another, the client return an > error. > > Do we have any plan to support cp/mv file cross namespace? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13274) RBF: Extend RouterRpcClient to use multiple sockets
[ https://issues.apache.org/jira/browse/HDFS-13274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13274: Parent Issue: HDFS-13891 (was: HDFS-12615) > RBF: Extend RouterRpcClient to use multiple sockets > --- > > Key: HDFS-13274 > URL: https://issues.apache.org/jira/browse/HDFS-13274 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > > HADOOP-13144 introduces the ability to create multiple connections for the > same user and use different sockets. The RouterRpcClient should use this > approach to get a better throughput. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13248) RBF: Namenode need to choose block location for the client
[ https://issues.apache.org/jira/browse/HDFS-13248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13248: Parent Issue: HDFS-13891 (was: HDFS-12615) > RBF: Namenode need to choose block location for the client > -- > > Key: HDFS-13248 > URL: https://issues.apache.org/jira/browse/HDFS-13248 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Weiwei Wu >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch, > clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg > > > When execute a put operation via router, the NameNode will choose block > location for the router, not for the real client. This will affect the file's > locality. > I think on both NameNode and Router, we should add a new addBlock method, or > add a parameter for the current addBlock method, to pass the real client > information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13255) RBF: Fail when try to remove mount point paths
[
https://issues.apache.org/jira/browse/HDFS-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-13255:
Parent Issue: HDFS-13891 (was: HDFS-12615)
> RBF: Fail when try to remove mount point paths
> --
>
> Key: HDFS-13255
> URL: https://issues.apache.org/jira/browse/HDFS-13255
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Weiwei Wu
>Priority: Major
>
> when delete a ns-fed path which include mount point paths, will issue a error.
> Need to delete each mount point path independently.
> Operation step:
> {code:java}
> [hadp@root]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> Source Destinations Owner Group Mode Quota/Usage
> /rm-test-all/rm-test-ns10 ns10->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> /rm-test-all/rm-test-ns2 ns1->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns2/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 101 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/NOTICE.txt
> -rw-r--r-- 3 hadp supergroup 1366 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/README.txt
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -rm -r hdfs://ns-fed/rm-test-all/
> rm: Failed to move to trash: hdfs://ns-fed/rm-test-all. Consider using
> -skipTrash option
> [hadp@root]$ hdfs dfs -rm -r -skipTrash hdfs://ns-fed/rm-test-all/
> rm: `hdfs://ns-fed/rm-test-all': Input/output error
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13443) RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries.
[
https://issues.apache.org/jira/browse/HDFS-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608500#comment-16608500
]
Hadoop QA commented on HDFS-13443:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HDFS-13443 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-13443 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12924546/HDFS-13443.011.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25009/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Update mount table cache immediately after changing (add/update/remove)
> mount table entries.
> -
>
> Key: HDFS-13443
> URL: https://issues.apache.org/jira/browse/HDFS-13443
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Major
> Labels: RBF
> Attachments: HDFS-13443-branch-2.001.patch,
> HDFS-13443-branch-2.002.patch, HDFS-13443.001.patch, HDFS-13443.002.patch,
> HDFS-13443.003.patch, HDFS-13443.004.patch, HDFS-13443.005.patch,
> HDFS-13443.006.patch, HDFS-13443.007.patch, HDFS-13443.008.patch,
> HDFS-13443.009.patch, HDFS-13443.010.patch, HDFS-13443.011.patch
>
>
> Currently mount table cache is updated periodically, by default cache is
> updated every minute. After change in mount table, user operations may still
> use old mount table. This is bit wrong.
> To update mount table cache, maybe we can do following
> * *Add refresh API in MountTableManager which will update mount table cache.*
> * *When there is a change in mount table entries, router admin server can
> update its cache and ask other routers to update their cache*. For example if
> there are three routers R1,R2,R3 in a cluster then add mount table entry API,
> at admin server side, will perform following sequence of action
> ## user submit add mount table entry request on R1
> ## R1 adds the mount table entry in state store
> ## R1 call refresh API on R2
> ## R1 calls refresh API on R3
> ## R1 directly freshest its cache
> ## Add mount table entry response send back to user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13245) RBF: State store DBMS implementation
[ https://issues.apache.org/jira/browse/HDFS-13245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13245: Parent Issue: HDFS-13891 (was: HDFS-12615) > RBF: State store DBMS implementation > > > Key: HDFS-13245 > URL: https://issues.apache.org/jira/browse/HDFS-13245 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: maobaolong >Assignee: Yiran Wu >Priority: Major > Attachments: HDFS-13245.001.patch, HDFS-13245.002.patch, > HDFS-13245.003.patch, HDFS-13245.004.patch, HDFS-13245.005.patch, > HDFS-13245.006.patch, HDFS-13245.007.patch, HDFS-13245.008.patch, > HDFS-13245.009.patch, HDFS-13245.010.patch, HDFS-13245.011.patch, > HDFS-13245.012.patch > > > Add a DBMS implementation for the State Store. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13278) RBF: Correct the logic of mount validate to avoid the bad mountPoint
[ https://issues.apache.org/jira/browse/HDFS-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608499#comment-16608499 ] Brahma Reddy Battula commented on HDFS-13278: - [~maobaolong] can you please confirm? > RBF: Correct the logic of mount validate to avoid the bad mountPoint > > > Key: HDFS-13278 > URL: https://issues.apache.org/jira/browse/HDFS-13278 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Affects Versions: 3.2.0 >Reporter: maobaolong >Priority: Major > Labels: RBF > > Correct the logic of mount validate to avoid the bad mountPoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13278) RBF: Correct the logic of mount validate to avoid the bad mountPoint
[ https://issues.apache.org/jira/browse/HDFS-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13278: Parent Issue: HDFS-13891 (was: HDFS-12615) > RBF: Correct the logic of mount validate to avoid the bad mountPoint > > > Key: HDFS-13278 > URL: https://issues.apache.org/jira/browse/HDFS-13278 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Affects Versions: 3.2.0 >Reporter: maobaolong >Priority: Major > Labels: RBF > > Correct the logic of mount validate to avoid the bad mountPoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13862) RBF: Router logs are not capturing few of the dfsrouteradmin commands
[ https://issues.apache.org/jira/browse/HDFS-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608498#comment-16608498 ] Ayush Saxena commented on HDFS-13862: - Thanx [~brahmareddy] for correcting.Got little confused in the previous comment. Have uploaded the patch v5 with the change as per ur comment. > RBF: Router logs are not capturing few of the dfsrouteradmin commands > - > > Key: HDFS-13862 > URL: https://issues.apache.org/jira/browse/HDFS-13862 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Soumyapn >Assignee: Ayush Saxena >Priority: Major > Labels: RBF > Attachments: HDFS-13862-01.patch, HDFS-13862-02.patch, > HDFS-13862-03.patch, HDFS-13862-04.patch, HDFS-13862-05.patch > > > Test Steps : > Below commands are not getting captured in the Router logs. > # Destination entry name in the add command. Log says "Added new mount point > /apps9 to resolver". > # Safemode enter|leave|get commands > # nameservice enable -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13443) RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries.
[ https://issues.apache.org/jira/browse/HDFS-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13443: Parent Issue: HDFS-13891 (was: HDFS-12615) > RBF: Update mount table cache immediately after changing (add/update/remove) > mount table entries. > - > > Key: HDFS-13443 > URL: https://issues.apache.org/jira/browse/HDFS-13443 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Mohammad Arshad >Assignee: Mohammad Arshad >Priority: Major > Labels: RBF > Attachments: HDFS-13443-branch-2.001.patch, > HDFS-13443-branch-2.002.patch, HDFS-13443.001.patch, HDFS-13443.002.patch, > HDFS-13443.003.patch, HDFS-13443.004.patch, HDFS-13443.005.patch, > HDFS-13443.006.patch, HDFS-13443.007.patch, HDFS-13443.008.patch, > HDFS-13443.009.patch, HDFS-13443.010.patch, HDFS-13443.011.patch > > > Currently mount table cache is updated periodically, by default cache is > updated every minute. After change in mount table, user operations may still > use old mount table. This is bit wrong. > To update mount table cache, maybe we can do following > * *Add refresh API in MountTableManager which will update mount table cache.* > * *When there is a change in mount table entries, router admin server can > update its cache and ask other routers to update their cache*. For example if > there are three routers R1,R2,R3 in a cluster then add mount table entry API, > at admin server side, will perform following sequence of action > ## user submit add mount table entry request on R1 > ## R1 adds the mount table entry in state store > ## R1 call refresh API on R2 > ## R1 calls refresh API on R3 > ## R1 directly freshest its cache > ## Add mount table entry response send back to user. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13470) RBF: Add Browse the Filesystem button to the UI
[ https://issues.apache.org/jira/browse/HDFS-13470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13470: Parent Issue: HDFS-13891 (was: HDFS-12615) > RBF: Add Browse the Filesystem button to the UI > --- > > Key: HDFS-13470 > URL: https://issues.apache.org/jira/browse/HDFS-13470 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13470.000.patch > > > After HDFS-12512 added WebHDFS, we can add the support to browse the > filesystem to the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13862) RBF: Router logs are not capturing few of the dfsrouteradmin commands
[ https://issues.apache.org/jira/browse/HDFS-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-13862: Attachment: HDFS-13862-05.patch > RBF: Router logs are not capturing few of the dfsrouteradmin commands > - > > Key: HDFS-13862 > URL: https://issues.apache.org/jira/browse/HDFS-13862 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Soumyapn >Assignee: Ayush Saxena >Priority: Major > Labels: RBF > Attachments: HDFS-13862-01.patch, HDFS-13862-02.patch, > HDFS-13862-03.patch, HDFS-13862-04.patch, HDFS-13862-05.patch > > > Test Steps : > Below commands are not getting captured in the Router logs. > # Destination entry name in the add command. Log says "Added new mount point > /apps9 to resolver". > # Safemode enter|leave|get commands > # nameservice enable -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13495) RBF: Support Router Admin REST API
[ https://issues.apache.org/jira/browse/HDFS-13495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13495: Parent Issue: HDFS-13891 (was: HDFS-12615) > RBF: Support Router Admin REST API > -- > > Key: HDFS-13495 > URL: https://issues.apache.org/jira/browse/HDFS-13495 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Mohammad Arshad >Assignee: Mohammad Arshad >Priority: Major > Labels: RBF > > This JIRA intends to add REST API support for all admin commands. Router > Admin REST APIs can be useful in managing the Routers from a central > management layer tool. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13507) RBF: Remove update functionality from routeradmin's add cmd
[ https://issues.apache.org/jira/browse/HDFS-13507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13507: Parent Issue: HDFS-13891 (was: HDFS-13815) > RBF: Remove update functionality from routeradmin's add cmd > --- > > Key: HDFS-13507 > URL: https://issues.apache.org/jira/browse/HDFS-13507 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei Yan >Assignee: Gang Li >Priority: Minor > Labels: incompatible > Attachments: HDFS-13507.000.patch, HDFS-13507.001.patch, > HDFS-13507.002.patch > > > Follow up the discussion in HDFS-13326. We should remove the "update" > functionality from routeradmin's add cmd, to make it consistent with RPC > calls. > Note that: this is an incompatible change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13507) RBF: Remove update functionality from routeradmin's add cmd
[ https://issues.apache.org/jira/browse/HDFS-13507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13507: Parent Issue: HDFS-13815 (was: HDFS-12615) > RBF: Remove update functionality from routeradmin's add cmd > --- > > Key: HDFS-13507 > URL: https://issues.apache.org/jira/browse/HDFS-13507 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei Yan >Assignee: Gang Li >Priority: Minor > Labels: incompatible > Attachments: HDFS-13507.000.patch, HDFS-13507.001.patch, > HDFS-13507.002.patch > > > Follow up the discussion in HDFS-13326. We should remove the "update" > functionality from routeradmin's add cmd, to make it consistent with RPC > calls. > Note that: this is an incompatible change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13577) RBF: Failed mount point operations, returns wrong exit code.
[
https://issues.apache.org/jira/browse/HDFS-13577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608496#comment-16608496
]
Brahma Reddy Battula commented on HDFS-13577:
-
HDFS-13815 is addressed this issue, can you guys cross check once..?
> RBF: Failed mount point operations, returns wrong exit code.
>
>
> Key: HDFS-13577
> URL: https://issues.apache.org/jira/browse/HDFS-13577
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Dibyendu Karmakar
>Priority: Major
> Labels: RBF
>
> If client is performed add mount point with some special character, mount
> point add is failed.
> And prints the message like
> {noformat}
> 18/05/17 09:58:34 DEBUG ipc.ProtobufRpcEngine: Call: addMountTableEntry took
> 19ms Cannot add mount point /testSpecialCharMountPointCreation/test/
> {noformat}
> In the above case it should return the exist code is non zero value.
> {code:java|title=RouterAdmin.java|borderStyle=solid}
> Exception debugException = null;
> exitCode = 0;
> try {
> if ("-add".equals(cmd)) {
> if (addMount(argv, i)) {
> System.out.println("Successfully added mount point " + argv[i]);
> }
> {code}
> we should handle this kind of cases also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13853) RBF: RouterAdmin update cmd is overwriting the entry not updating the existing
[
https://issues.apache.org/jira/browse/HDFS-13853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-13853:
Parent Issue: HDFS-13891 (was: HDFS-12615)
> RBF: RouterAdmin update cmd is overwriting the entry not updating the existing
> --
>
> Key: HDFS-13853
> URL: https://issues.apache.org/jira/browse/HDFS-13853
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Dibyendu Karmakar
>Assignee: Dibyendu Karmakar
>Priority: Major
>
> {code:java}
> // Create a new entry
> Map destMap = new LinkedHashMap<>();
> for (String ns : nss) {
> destMap.put(ns, dest);
> }
> MountTable newEntry = MountTable.newInstance(mount, destMap);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12615) Router-based HDFS federation phase 2
[ https://issues.apache.org/jira/browse/HDFS-12615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608492#comment-16608492 ] Brahma Reddy Battula commented on HDFS-12615: - Thanks [~elgoiri] and [~anu] .. will move out and close this. > Router-based HDFS federation phase 2 > > > Key: HDFS-12615 > URL: https://issues.apache.org/jira/browse/HDFS-12615 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Labels: RBF > > This umbrella JIRA tracks set of improvements over the Router-based HDFS > federation (HDFS-10467). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13862) RBF: Router logs are not capturing few of the dfsrouteradmin commands
[ https://issues.apache.org/jira/browse/HDFS-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608490#comment-16608490 ] Brahma Reddy Battula commented on HDFS-13862: - Yes, but commands are different right.. you can keep the format same.. "dfsadmin" will not work with router right..? > RBF: Router logs are not capturing few of the dfsrouteradmin commands > - > > Key: HDFS-13862 > URL: https://issues.apache.org/jira/browse/HDFS-13862 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Soumyapn >Assignee: Ayush Saxena >Priority: Major > Labels: RBF > Attachments: HDFS-13862-01.patch, HDFS-13862-02.patch, > HDFS-13862-03.patch, HDFS-13862-04.patch > > > Test Steps : > Below commands are not getting captured in the Router logs. > # Destination entry name in the add command. Log says "Added new mount point > /apps9 to resolver". > # Safemode enter|leave|get commands > # nameservice enable -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13862) RBF: Router logs are not capturing few of the dfsrouteradmin commands
[
https://issues.apache.org/jira/browse/HDFS-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608482#comment-16608482
]
Ayush Saxena edited comment on HDFS-13862 at 9/9/18 4:09 PM:
-
Thanx [~brahmareddy] for the comment.
As per the Discussion we need to keep it in sync with the namenode?
In FsNamesystem.java where this is handled for namenode side safemode it is
like this only.Pls Correct if I am wrong. :)
{code:java}
String getSafeModeTip() {
String cmd = "Use \"hdfs dfsadmin -safemode leave\" to turn safe mode
off.";{code}
was (Author: ayushtkn):
Thanx [~brahmareddy] for the comment.
> RBF: Router logs are not capturing few of the dfsrouteradmin commands
> -
>
> Key: HDFS-13862
> URL: https://issues.apache.org/jira/browse/HDFS-13862
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Soumyapn
>Assignee: Ayush Saxena
>Priority: Major
> Labels: RBF
> Attachments: HDFS-13862-01.patch, HDFS-13862-02.patch,
> HDFS-13862-03.patch, HDFS-13862-04.patch
>
>
> Test Steps :
> Below commands are not getting captured in the Router logs.
> # Destination entry name in the add command. Log says "Added new mount point
> /apps9 to resolver".
> # Safemode enter|leave|get commands
> # nameservice enable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-399) Handle pipeline discovery on SCM restart.
[
https://issues.apache.org/jira/browse/HDDS-399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608484#comment-16608484
]
Hadoop QA commented on HDDS-399:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 23m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
4m 3s{color} | {color:orange} root: The patch generated 5 new + 20 unchanged -
1 fixed = 25 total (was 21) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
26s{color} | {color:red} hadoop-hdds/common generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
56s{color} | {color:red} hadoop-hdds/server-scm generated 4 new + 0 unchanged -
0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
6s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
44s{color} | {color:green} server-scm in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 50s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
50s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdds/common |
| |
[jira] [Assigned] (HDDS-420) putKey failing with KEY_ALLOCATION_ERROR
[
https://issues.apache.org/jira/browse/HDDS-420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee reassigned HDDS-420:
Assignee: Shashikant Banerjee
> putKey failing with KEY_ALLOCATION_ERROR
>
>
> Key: HDDS-420
> URL: https://issues.apache.org/jira/browse/HDDS-420
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Nilotpal Nandi
>Assignee: Shashikant Banerjee
>Priority: Critical
> Fix For: 0.2.1
>
>
> Here are the commands run :
> {noformat}
> [root@ctr-e138-1518143905142-468367-01-02 bin]# ./ozone oz -putKey
> /fs-volume/fs-bucket/nn1 -file /etc/passwd
> 2018-09-09 15:39:31,131 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Create key failed, error:KEY_ALLOCATION_ERROR
> [root@ctr-e138-1518143905142-468367-01-02 bin]#
> [root@ctr-e138-1518143905142-468367-01-02 bin]# ./ozone fs -copyFromLocal
> /etc/passwd /
> 2018-09-09 15:40:16,879 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2018-09-09 15:40:23,632 [main] ERROR - Try to allocate more blocks for write
> failed, already allocated 0 blocks for this write.
> copyFromLocal: Message missing required fields: keyLocation
> [root@ctr-e138-1518143905142-468367-01-02 bin]# ./ozone oz -putKey
> /fs-volume/fs-bucket/nn2 -file /etc/passwd
> 2018-09-09 15:44:55,912 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Create key failed, error:KEY_ALLOCATION_ERROR{noformat}
>
> hadoop version :
> ---
> {noformat}
> [root@ctr-e138-1518143905142-468367-01-02 bin]# ./hadoop version
> Hadoop 3.2.0-SNAPSHOT
> Source code repository git://git.apache.org/hadoop.git -r
> bf8a1750e99cfbfa76021ce51b6514c74c06f498
> Compiled by root on 2018-09-08T10:22Z
> Compiled with protoc 2.5.0
> From source with checksum c5bbb375aed8edabd89c377af83189d
> This command was run using
> /root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT.jar{noformat}
>
> scm log :
> ---
> {noformat}
> 2018-09-09 15:45:00,907 INFO
> org.apache.hadoop.hdds.scm.pipelines.ratis.RatisManagerImpl: Allocating a new
> ratis pipeline of size: 3 id: pipelineId=f210716d-ba7b-4adf-91d6-da286e5fd010
> 2018-09-09 15:45:00,973 INFO org.apache.ratis.conf.ConfUtils: raft.rpc.type =
> GRPC (default)
> 2018-09-09 15:45:01,007 INFO org.apache.ratis.conf.ConfUtils:
> raft.grpc.message.size.max = 33554432 (custom)
> 2018-09-09 15:45:01,011 INFO org.apache.ratis.conf.ConfUtils:
> raft.client.rpc.retryInterval = 300 ms (default)
> 2018-09-09 15:45:01,012 INFO org.apache.ratis.conf.ConfUtils:
> raft.client.async.outstanding-requests.max = 100 (default)
> 2018-09-09 15:45:01,012 INFO org.apache.ratis.conf.ConfUtils:
> raft.client.async.scheduler-threads = 3 (default)
> 2018-09-09 15:45:01,020 INFO org.apache.ratis.conf.ConfUtils:
> raft.grpc.flow.control.window = 1MB (=1048576) (default)
> 2018-09-09 15:45:01,020 INFO org.apache.ratis.conf.ConfUtils:
> raft.grpc.message.size.max = 33554432 (custom)
> 2018-09-09 15:45:01,102 INFO org.apache.ratis.conf.ConfUtils:
> raft.client.rpc.request.timeout = 3000 ms (default)
> 2018-09-09 15:45:01,667 ERROR org.apache.hadoop.hdds.scm.XceiverClientRatis:
> Failed to reinitialize
> RaftPeer:bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9:172.27.12.96:9858 datanode:
> bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9{ip: 172.27.12.96, host:
> ctr-e138-1518143905142-468367-01-07.hwx.site}
> org.apache.ratis.protocol.GroupMismatchException:
> bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9: The group (group-7347726F7570) of
> client-409D68EB500F does not match the group (group-2041ABBEE452) of the
> server bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.ratis.util.ReflectionUtils.instantiateException(ReflectionUtils.java:222)
> at
> org.apache.ratis.grpc.RaftGrpcUtil.tryUnwrapException(RaftGrpcUtil.java:79)
> at org.apache.ratis.grpc.RaftGrpcUtil.unwrapException(RaftGrpcUtil.java:67)
> at
> org.apache.ratis.grpc.client.RaftClientProtocolClient.blockingCall(RaftClientProtocolClient.java:127)
> at
>
[jira] [Commented] (HDFS-13862) RBF: Router logs are not capturing few of the dfsrouteradmin commands
[ https://issues.apache.org/jira/browse/HDFS-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608482#comment-16608482 ] Ayush Saxena commented on HDFS-13862: - Thanx [~brahmareddy] for the comment. > RBF: Router logs are not capturing few of the dfsrouteradmin commands > - > > Key: HDFS-13862 > URL: https://issues.apache.org/jira/browse/HDFS-13862 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Soumyapn >Assignee: Ayush Saxena >Priority: Major > Labels: RBF > Attachments: HDFS-13862-01.patch, HDFS-13862-02.patch, > HDFS-13862-03.patch, HDFS-13862-04.patch > > > Test Steps : > Below commands are not getting captured in the Router logs. > # Destination entry name in the add command. Log says "Added new mount point > /apps9 to resolver". > # Safemode enter|leave|get commands > # nameservice enable -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-419) ChunkInputStream bulk read api does not read from all the chunks
[ https://issues.apache.org/jira/browse/HDDS-419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608480#comment-16608480 ] Shashikant Banerjee commented on HDDS-419: -- Thanks [~msingh] for working on this. patch v1 looks good to me. +1. > ChunkInputStream bulk read api does not read from all the chunks > > > Key: HDDS-419 > URL: https://issues.apache.org/jira/browse/HDDS-419 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.2.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Blocker > Fix For: 0.2.1 > > Attachments: HDDS-419.001.patch > > > After enabling of bulk reads with HDDS-408, testDataValidate started failing > because the bulk read api does not read all the chunks from the block. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13862) RBF: Router logs are not capturing few of the dfsrouteradmin commands
[ https://issues.apache.org/jira/browse/HDFS-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608479#comment-16608479 ] Brahma Reddy Battula commented on HDFS-13862: - [~ayushtkn] thanks for updating the patch. you need to change following to "Usage: hdfs routeradmin :". |88|+ "Use \"hdfs dfsadmin -safemode leave\" to turn safe mode off.");| > RBF: Router logs are not capturing few of the dfsrouteradmin commands > - > > Key: HDFS-13862 > URL: https://issues.apache.org/jira/browse/HDFS-13862 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Soumyapn >Assignee: Ayush Saxena >Priority: Major > Labels: RBF > Attachments: HDFS-13862-01.patch, HDFS-13862-02.patch, > HDFS-13862-03.patch, HDFS-13862-04.patch > > > Test Steps : > Below commands are not getting captured in the Router logs. > # Destination entry name in the add command. Log says "Added new mount point > /apps9 to resolver". > # Safemode enter|leave|get commands > # nameservice enable -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-420) putKey failing with KEY_ALLOCATION_ERROR
Nilotpal Nandi created HDDS-420:
---
Summary: putKey failing with KEY_ALLOCATION_ERROR
Key: HDDS-420
URL: https://issues.apache.org/jira/browse/HDDS-420
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: Ozone Manager
Reporter: Nilotpal Nandi
Fix For: 0.2.1
Here are the commands run :
{noformat}
[root@ctr-e138-1518143905142-468367-01-02 bin]# ./ozone oz -putKey
/fs-volume/fs-bucket/nn1 -file /etc/passwd
2018-09-09 15:39:31,131 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Create key failed, error:KEY_ALLOCATION_ERROR
[root@ctr-e138-1518143905142-468367-01-02 bin]#
[root@ctr-e138-1518143905142-468367-01-02 bin]# ./ozone fs -copyFromLocal
/etc/passwd /
2018-09-09 15:40:16,879 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2018-09-09 15:40:23,632 [main] ERROR - Try to allocate more blocks for write
failed, already allocated 0 blocks for this write.
copyFromLocal: Message missing required fields: keyLocation
[root@ctr-e138-1518143905142-468367-01-02 bin]# ./ozone oz -putKey
/fs-volume/fs-bucket/nn2 -file /etc/passwd
2018-09-09 15:44:55,912 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Create key failed, error:KEY_ALLOCATION_ERROR{noformat}
hadoop version :
---
{noformat}
[root@ctr-e138-1518143905142-468367-01-02 bin]# ./hadoop version
Hadoop 3.2.0-SNAPSHOT
Source code repository git://git.apache.org/hadoop.git -r
bf8a1750e99cfbfa76021ce51b6514c74c06f498
Compiled by root on 2018-09-08T10:22Z
Compiled with protoc 2.5.0
>From source with checksum c5bbb375aed8edabd89c377af83189d
This command was run using
/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT.jar{noformat}
scm log :
---
{noformat}
2018-09-09 15:45:00,907 INFO
org.apache.hadoop.hdds.scm.pipelines.ratis.RatisManagerImpl: Allocating a new
ratis pipeline of size: 3 id: pipelineId=f210716d-ba7b-4adf-91d6-da286e5fd010
2018-09-09 15:45:00,973 INFO org.apache.ratis.conf.ConfUtils: raft.rpc.type =
GRPC (default)
2018-09-09 15:45:01,007 INFO org.apache.ratis.conf.ConfUtils:
raft.grpc.message.size.max = 33554432 (custom)
2018-09-09 15:45:01,011 INFO org.apache.ratis.conf.ConfUtils:
raft.client.rpc.retryInterval = 300 ms (default)
2018-09-09 15:45:01,012 INFO org.apache.ratis.conf.ConfUtils:
raft.client.async.outstanding-requests.max = 100 (default)
2018-09-09 15:45:01,012 INFO org.apache.ratis.conf.ConfUtils:
raft.client.async.scheduler-threads = 3 (default)
2018-09-09 15:45:01,020 INFO org.apache.ratis.conf.ConfUtils:
raft.grpc.flow.control.window = 1MB (=1048576) (default)
2018-09-09 15:45:01,020 INFO org.apache.ratis.conf.ConfUtils:
raft.grpc.message.size.max = 33554432 (custom)
2018-09-09 15:45:01,102 INFO org.apache.ratis.conf.ConfUtils:
raft.client.rpc.request.timeout = 3000 ms (default)
2018-09-09 15:45:01,667 ERROR org.apache.hadoop.hdds.scm.XceiverClientRatis:
Failed to reinitialize
RaftPeer:bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9:172.27.12.96:9858 datanode:
bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9{ip: 172.27.12.96, host:
ctr-e138-1518143905142-468367-01-07.hwx.site}
org.apache.ratis.protocol.GroupMismatchException:
bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9: The group (group-7347726F7570) of
client-409D68EB500F does not match the group (group-2041ABBEE452) of the server
bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.ratis.util.ReflectionUtils.instantiateException(ReflectionUtils.java:222)
at org.apache.ratis.grpc.RaftGrpcUtil.tryUnwrapException(RaftGrpcUtil.java:79)
at org.apache.ratis.grpc.RaftGrpcUtil.unwrapException(RaftGrpcUtil.java:67)
at
org.apache.ratis.grpc.client.RaftClientProtocolClient.blockingCall(RaftClientProtocolClient.java:127)
at
org.apache.ratis.grpc.client.RaftClientProtocolClient.reinitialize(RaftClientProtocolClient.java:102)
at
org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:77)
at
org.apache.ratis.client.impl.RaftClientImpl.sendRequest(RaftClientImpl.java:302)
at
org.apache.ratis.client.impl.RaftClientImpl.reinitialize(RaftClientImpl.java:216)
at
org.apache.hadoop.hdds.scm.XceiverClientRatis.reinitialize(XceiverClientRatis.java:163)
at
org.apache.hadoop.hdds.scm.XceiverClientRatis.reinitialize(XceiverClientRatis.java:133)
[jira] [Updated] (HDDS-419) ChunkInputStream bulk read api does not read from all the chunks
[ https://issues.apache.org/jira/browse/HDDS-419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-419: --- Attachment: HDDS-419.001.patch > ChunkInputStream bulk read api does not read from all the chunks > > > Key: HDDS-419 > URL: https://issues.apache.org/jira/browse/HDDS-419 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.2.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Blocker > Fix For: 0.2.1 > > Attachments: HDDS-419.001.patch > > > After enabling of bulk reads with HDDS-408, testDataValidate started failing > because the bulk read api does not read all the chunks from the block. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-419) ChunkInputStream bulk read api does not read from all the chunks
Mukul Kumar Singh created HDDS-419: -- Summary: ChunkInputStream bulk read api does not read from all the chunks Key: HDDS-419 URL: https://issues.apache.org/jira/browse/HDDS-419 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Affects Versions: 0.2.1 Reporter: Mukul Kumar Singh Assignee: Mukul Kumar Singh Fix For: 0.2.1 After enabling of bulk reads with HDDS-408, testDataValidate started failing because the bulk read api does not read all the chunks from the block. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-399) Handle pipeline discovery on SCM restart.
[ https://issues.apache.org/jira/browse/HDDS-399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608454#comment-16608454 ] Mukul Kumar Singh commented on HDDS-399: Thanks for the review [~anu]. Patch v3 fixes the review comments. > Handle pipeline discovery on SCM restart. > - > > Key: HDDS-399 > URL: https://issues.apache.org/jira/browse/HDDS-399 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.2.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Blocker > Fix For: 0.2.1 > > Attachments: HDDS-399.001.patch, HDDS-399.002.patch, > HDDS-399.003.patch > > > On SCM restart, as part on node registration, SCM should find out the list on > open pipeline on the node. Once all the nodes of the pipeline have reported > back, they should be added as active pipelines for further allocations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-399) Handle pipeline discovery on SCM restart.
[ https://issues.apache.org/jira/browse/HDDS-399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-399: --- Attachment: HDDS-399.003.patch > Handle pipeline discovery on SCM restart. > - > > Key: HDDS-399 > URL: https://issues.apache.org/jira/browse/HDDS-399 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.2.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Blocker > Fix For: 0.2.1 > > Attachments: HDDS-399.001.patch, HDDS-399.002.patch, > HDDS-399.003.patch > > > On SCM restart, as part on node registration, SCM should find out the list on > open pipeline on the node. Once all the nodes of the pipeline have reported > back, they should be added as active pipelines for further allocations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-399) Handle pipeline discovery on SCM restart.
[ https://issues.apache.org/jira/browse/HDDS-399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-399: --- Attachment: (was: HDDS-399.003.patch) > Handle pipeline discovery on SCM restart. > - > > Key: HDDS-399 > URL: https://issues.apache.org/jira/browse/HDDS-399 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.2.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Blocker > Fix For: 0.2.1 > > Attachments: HDDS-399.001.patch, HDDS-399.002.patch, > HDDS-399.003.patch > > > On SCM restart, as part on node registration, SCM should find out the list on > open pipeline on the node. Once all the nodes of the pipeline have reported > back, they should be added as active pipelines for further allocations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-399) Handle pipeline discovery on SCM restart.
[ https://issues.apache.org/jira/browse/HDDS-399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-399: --- Attachment: HDDS-399.003.patch > Handle pipeline discovery on SCM restart. > - > > Key: HDDS-399 > URL: https://issues.apache.org/jira/browse/HDDS-399 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.2.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Blocker > Fix For: 0.2.1 > > Attachments: HDDS-399.001.patch, HDDS-399.002.patch, > HDDS-399.003.patch > > > On SCM restart, as part on node registration, SCM should find out the list on > open pipeline on the node. Once all the nodes of the pipeline have reported > back, they should be added as active pipelines for further allocations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
