[jira] [Commented] (HDDS-1109) Setup Failover Proxy Provider for client
[ https://issues.apache.org/jira/browse/HDDS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772730#comment-16772730 ] Yiqun Lin commented on HDDS-1109: - Looks great now, +1 once checkstyle issues fixed. > Setup Failover Proxy Provider for client > > > Key: HDDS-1109 > URL: https://issues.apache.org/jira/browse/HDDS-1109 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HDDS-1109.001.patch, HDDS-1109.002.patch > > > If a client sends a request to a OM follower, it will get a > NotLeaderException. The client should then keep trying the request on other > OMs till it finds the leader OM. Client should cache the information about > current OM leader node. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14216) NullPointerException happens in NamenodeWebHdfs
[
https://issues.apache.org/jira/browse/HDFS-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772693#comment-16772693
]
lujie commented on HDFS-14216:
--
Odd test failure, my local test works well
> NullPointerException happens in NamenodeWebHdfs
> ---
>
> Key: HDFS-14216
> URL: https://issues.apache.org/jira/browse/HDFS-14216
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: lujie
>Priority: Critical
> Attachments: HDFS-14216_1.patch, HDFS-14216_2.patch,
> HDFS-14216_3.patch, HDFS-14216_4.patch, hadoop-hires-namenode-hadoop11.log
>
>
> workload
> {code:java}
> curl -i -X PUT -T $HOMEPARH/test.txt
> "http://hadoop1:9870/webhdfs/v1/input?op=CREATE=hadoop2;
> {code}
> the method
> {code:java}
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.chooseDatanode(String
> excludeDatanodes){
> HashSet excludes = new HashSet();
> if (excludeDatanodes != null) {
>for (String host : StringUtils
> .getTrimmedStringCollection(excludeDatanodes)) {
> int idx = host.indexOf(":");
>if (idx != -1) {
> excludes.add(bm.getDatanodeManager().getDatanodeByXferAddr(
>host.substring(0, idx), Integer.parseInt(host.substring(idx +
> 1;
>} else {
>
> excludes.add(bm.getDatanodeManager().getDatanodeByHost(host));//line280
>}
> }
> }
> }
> {code}
> when datanode(e.g.hadoop2) is {color:#d04437}just wiped before
> line280{color}, or{color:#33}
> {color}{color:#ff}we{color}{color:#ff} give the wrong DN
> name{color}*,*then bm.getDatanodeManager().getDatanodeByHost(host) will
> return null, *_excludes_* *containes null*. while *_excludes_* are used
> later, NPE happens:
> {code:java}
> java.lang.NullPointerException
> at org.apache.hadoop.net.NodeBase.getPath(NodeBase.java:113)
> at
> org.apache.hadoop.net.NetworkTopology.countNumOfAvailableNodes(NetworkTopology.java:672)
> at
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:533)
> at
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:491)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.chooseDatanode(NamenodeWebHdfsMethods.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.redirectURI(NamenodeWebHdfsMethods.java:384)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.put(NamenodeWebHdfsMethods.java:652)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$2.run(NamenodeWebHdfsMethods.java:600)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$2.run(NamenodeWebHdfsMethods.java:597)
> at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:73)
> at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:30)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830)
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14235) Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon
[
https://issues.apache.org/jira/browse/HDFS-14235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ranith Sardar updated HDFS-14235:
-
Attachment: HDFS-14235.003.patch
> Handle ArrayIndexOutOfBoundsException in
> DataNodeDiskMetrics#slowDiskDetectionDaemon
> -
>
> Key: HDFS-14235
> URL: https://issues.apache.org/jira/browse/HDFS-14235
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Surendra Singh Lilhore
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-14235.000.patch, HDFS-14235.001.patch,
> HDFS-14235.002.patch, HDFS-14235.003.patch, NPE.png, exception.png
>
>
> below code throwing exception because {{volumeIterator.next()}} called two
> time without checking hashNext().
> {code:java}
> while (volumeIterator.hasNext()) {
> FsVolumeSpi volume = volumeIterator.next();
> DataNodeVolumeMetrics metrics = volumeIterator.next().getMetrics();
> String volumeName = volume.getBaseURI().getPath();
> metadataOpStats.put(volumeName,
> metrics.getMetadataOperationMean());
> readIoStats.put(volumeName, metrics.getReadIoMean());
> writeIoStats.put(volumeName, metrics.getWriteIoMean());
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14216) NullPointerException happens in NamenodeWebHdfs
[
https://issues.apache.org/jira/browse/HDFS-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772667#comment-16772667
]
Hadoop QA commented on HDFS-14216:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 156 unchanged - 1 fixed = 156 total (was 157) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 50s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14216 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959364/HDFS-14216_4.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 3730420047be 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1d30fd9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26264/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26264/testReport/ |
| Max. process+thread count | 3512 (vs. ulimit of 1) |
| modules | C:
[jira] [Commented] (HDFS-14254) RBF: Getfacl gives a wrong acl entries when the order of the mount table set to HASH_ALL or RANDOM
[
https://issues.apache.org/jira/browse/HDFS-14254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772648#comment-16772648
]
Hadoop QA commented on HDFS-14254:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
16s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 19s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRouterAllResolver |
| | hadoop.hdfs.server.federation.router.TestRouterMountTable |
| | hadoop.hdfs.server.federation.router.TestRouterQuota |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14254 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959365/HDFS-14254-HDFS-13891.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d9157d5b08f9 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / f94b6e3 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26265/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26265/testReport/ |
| Max. process+thread count | 1381 (vs. ulimit of 1) |
| modules | C:
[jira] [Updated] (HDDS-1095) OzoneManager#openKey should do multiple block allocations in a single SCM rpc call
[ https://issues.apache.org/jira/browse/HDDS-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-1095: Attachment: HDDS-1095.002.patch > OzoneManager#openKey should do multiple block allocations in a single SCM rpc > call > -- > > Key: HDDS-1095 > URL: https://issues.apache.org/jira/browse/HDDS-1095 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1095.001.patch, HDDS-1095.002.patch > > > Currently in KeyManagerImpl#openKey, for a large key allocation, multiple > blocks are allocated in different rpc calls. If the key length is already > known, then multiple blocks can be allocated in one rpc call to SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1095) OzoneManager#openKey should do multiple block allocations in a single SCM rpc call
[ https://issues.apache.org/jira/browse/HDDS-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772645#comment-16772645 ] Mukul Kumar Singh commented on HDDS-1095: - Thanks for the review [~bharatviswa]. The comments are incorporated in v2 patch. > OzoneManager#openKey should do multiple block allocations in a single SCM rpc > call > -- > > Key: HDDS-1095 > URL: https://issues.apache.org/jira/browse/HDDS-1095 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1095.001.patch, HDDS-1095.002.patch > > > Currently in KeyManagerImpl#openKey, for a large key allocation, multiple > blocks are allocated in different rpc calls. If the key length is already > known, then multiple blocks can be allocated in one rpc call to SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14235) Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon
[
https://issues.apache.org/jira/browse/HDFS-14235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772642#comment-16772642
]
Surendra Singh Lilhore commented on HDFS-14235:
---
Pls fix the checkstyle warnings.
> Handle ArrayIndexOutOfBoundsException in
> DataNodeDiskMetrics#slowDiskDetectionDaemon
> -
>
> Key: HDFS-14235
> URL: https://issues.apache.org/jira/browse/HDFS-14235
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Surendra Singh Lilhore
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-14235.000.patch, HDFS-14235.001.patch,
> HDFS-14235.002.patch, NPE.png, exception.png
>
>
> below code throwing exception because {{volumeIterator.next()}} called two
> time without checking hashNext().
> {code:java}
> while (volumeIterator.hasNext()) {
> FsVolumeSpi volume = volumeIterator.next();
> DataNodeVolumeMetrics metrics = volumeIterator.next().getMetrics();
> String volumeName = volume.getBaseURI().getPath();
> metadataOpStats.put(volumeName,
> metrics.getMetadataOperationMean());
> readIoStats.put(volumeName, metrics.getReadIoMean());
> writeIoStats.put(volumeName, metrics.getWriteIoMean());
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14130) Make ZKFC ObserverNode aware
[
https://issues.apache.org/jira/browse/HDFS-14130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772639#comment-16772639
]
Hadoop QA commented on HDFS-14130:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 58s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
3s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 3s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}203m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14130 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959352/HDFS-14130.009.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a57d1ef6a882 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 51950f1 |
| maven | version:
[jira] [Updated] (HDFS-14254) RBF: Getfacl gives a wrong acl entries when the order of the mount table set to HASH_ALL or RANDOM
[ https://issues.apache.org/jira/browse/HDFS-14254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ranith Sardar updated HDFS-14254: - Attachment: HDFS-14254-HDFS-13891.001.patch > RBF: Getfacl gives a wrong acl entries when the order of the mount table set > to HASH_ALL or RANDOM > -- > > Key: HDFS-14254 > URL: https://issues.apache.org/jira/browse/HDFS-14254 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Shubham Dewan >Assignee: Ranith Sardar >Priority: Major > Attachments: HDFS-14254-HDFS-13891.000.patch, > HDFS-14254-HDFS-13891.001.patch > > > ACL entries are missing when Order is set to HASH_ALL or RANDOM -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14081) hdfs dfsadmin -metasave metasave_test results NPE
[
https://issues.apache.org/jira/browse/HDFS-14081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772604#comment-16772604
]
Hadoop QA commented on HDFS-14081:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 1s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 510 unchanged - 0 fixed = 514 total (was 510) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 50s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 11s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14081 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959348/HDFS-14081.009.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7eb6f49e076a 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 51950f1 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Comment Edited] (HDFS-14216) NullPointerException happens in NamenodeWebHdfs
[
https://issues.apache.org/jira/browse/HDFS-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772605#comment-16772605
]
lujie edited comment on HDFS-14216 at 2/20/19 4:18 AM:
---
Hi [~xkrogen]
thanks for you nice review, I have changed the patch as your suggestion! See
[^HDFS-14216_4.patch]
was (Author: xiaoheipangzi):
Hi [~xkrogen]
thanks for you nice review, I have changed the patch as your suggestion! See
[^HDFS-14216_4.patch] [^HDFS-14216_4.patch]|[^HDFS-14216_4.patch]]
> NullPointerException happens in NamenodeWebHdfs
> ---
>
> Key: HDFS-14216
> URL: https://issues.apache.org/jira/browse/HDFS-14216
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: lujie
>Priority: Critical
> Attachments: HDFS-14216_1.patch, HDFS-14216_2.patch,
> HDFS-14216_3.patch, HDFS-14216_4.patch, hadoop-hires-namenode-hadoop11.log
>
>
> workload
> {code:java}
> curl -i -X PUT -T $HOMEPARH/test.txt
> "http://hadoop1:9870/webhdfs/v1/input?op=CREATE=hadoop2;
> {code}
> the method
> {code:java}
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.chooseDatanode(String
> excludeDatanodes){
> HashSet excludes = new HashSet();
> if (excludeDatanodes != null) {
>for (String host : StringUtils
> .getTrimmedStringCollection(excludeDatanodes)) {
> int idx = host.indexOf(":");
>if (idx != -1) {
> excludes.add(bm.getDatanodeManager().getDatanodeByXferAddr(
>host.substring(0, idx), Integer.parseInt(host.substring(idx +
> 1;
>} else {
>
> excludes.add(bm.getDatanodeManager().getDatanodeByHost(host));//line280
>}
> }
> }
> }
> {code}
> when datanode(e.g.hadoop2) is {color:#d04437}just wiped before
> line280{color}, or{color:#33}
> {color}{color:#ff}we{color}{color:#ff} give the wrong DN
> name{color}*,*then bm.getDatanodeManager().getDatanodeByHost(host) will
> return null, *_excludes_* *containes null*. while *_excludes_* are used
> later, NPE happens:
> {code:java}
> java.lang.NullPointerException
> at org.apache.hadoop.net.NodeBase.getPath(NodeBase.java:113)
> at
> org.apache.hadoop.net.NetworkTopology.countNumOfAvailableNodes(NetworkTopology.java:672)
> at
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:533)
> at
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:491)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.chooseDatanode(NamenodeWebHdfsMethods.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.redirectURI(NamenodeWebHdfsMethods.java:384)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.put(NamenodeWebHdfsMethods.java:652)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$2.run(NamenodeWebHdfsMethods.java:600)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$2.run(NamenodeWebHdfsMethods.java:597)
> at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:73)
> at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:30)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830)
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14216) NullPointerException happens in NamenodeWebHdfs
[
https://issues.apache.org/jira/browse/HDFS-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772605#comment-16772605
]
lujie commented on HDFS-14216:
--
Hi [~xkrogen]
thanks for you nice review, I have changed the patch as your suggestion! See
[^HDFS-14216_4.patch] [^HDFS-14216_4.patch]|[^HDFS-14216_4.patch]]
> NullPointerException happens in NamenodeWebHdfs
> ---
>
> Key: HDFS-14216
> URL: https://issues.apache.org/jira/browse/HDFS-14216
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: lujie
>Priority: Critical
> Attachments: HDFS-14216_1.patch, HDFS-14216_2.patch,
> HDFS-14216_3.patch, HDFS-14216_4.patch, hadoop-hires-namenode-hadoop11.log
>
>
> workload
> {code:java}
> curl -i -X PUT -T $HOMEPARH/test.txt
> "http://hadoop1:9870/webhdfs/v1/input?op=CREATE=hadoop2;
> {code}
> the method
> {code:java}
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.chooseDatanode(String
> excludeDatanodes){
> HashSet excludes = new HashSet();
> if (excludeDatanodes != null) {
>for (String host : StringUtils
> .getTrimmedStringCollection(excludeDatanodes)) {
> int idx = host.indexOf(":");
>if (idx != -1) {
> excludes.add(bm.getDatanodeManager().getDatanodeByXferAddr(
>host.substring(0, idx), Integer.parseInt(host.substring(idx +
> 1;
>} else {
>
> excludes.add(bm.getDatanodeManager().getDatanodeByHost(host));//line280
>}
> }
> }
> }
> {code}
> when datanode(e.g.hadoop2) is {color:#d04437}just wiped before
> line280{color}, or{color:#33}
> {color}{color:#ff}we{color}{color:#ff} give the wrong DN
> name{color}*,*then bm.getDatanodeManager().getDatanodeByHost(host) will
> return null, *_excludes_* *containes null*. while *_excludes_* are used
> later, NPE happens:
> {code:java}
> java.lang.NullPointerException
> at org.apache.hadoop.net.NodeBase.getPath(NodeBase.java:113)
> at
> org.apache.hadoop.net.NetworkTopology.countNumOfAvailableNodes(NetworkTopology.java:672)
> at
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:533)
> at
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:491)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.chooseDatanode(NamenodeWebHdfsMethods.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.redirectURI(NamenodeWebHdfsMethods.java:384)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.put(NamenodeWebHdfsMethods.java:652)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$2.run(NamenodeWebHdfsMethods.java:600)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$2.run(NamenodeWebHdfsMethods.java:597)
> at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:73)
> at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:30)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830)
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14216) NullPointerException happens in NamenodeWebHdfs
[
https://issues.apache.org/jira/browse/HDFS-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated HDFS-14216:
-
Attachment: HDFS-14216_4.patch
> NullPointerException happens in NamenodeWebHdfs
> ---
>
> Key: HDFS-14216
> URL: https://issues.apache.org/jira/browse/HDFS-14216
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: lujie
>Priority: Critical
> Attachments: HDFS-14216_1.patch, HDFS-14216_2.patch,
> HDFS-14216_3.patch, HDFS-14216_4.patch, hadoop-hires-namenode-hadoop11.log
>
>
> workload
> {code:java}
> curl -i -X PUT -T $HOMEPARH/test.txt
> "http://hadoop1:9870/webhdfs/v1/input?op=CREATE=hadoop2;
> {code}
> the method
> {code:java}
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.chooseDatanode(String
> excludeDatanodes){
> HashSet excludes = new HashSet();
> if (excludeDatanodes != null) {
>for (String host : StringUtils
> .getTrimmedStringCollection(excludeDatanodes)) {
> int idx = host.indexOf(":");
>if (idx != -1) {
> excludes.add(bm.getDatanodeManager().getDatanodeByXferAddr(
>host.substring(0, idx), Integer.parseInt(host.substring(idx +
> 1;
>} else {
>
> excludes.add(bm.getDatanodeManager().getDatanodeByHost(host));//line280
>}
> }
> }
> }
> {code}
> when datanode(e.g.hadoop2) is {color:#d04437}just wiped before
> line280{color}, or{color:#33}
> {color}{color:#ff}we{color}{color:#ff} give the wrong DN
> name{color}*,*then bm.getDatanodeManager().getDatanodeByHost(host) will
> return null, *_excludes_* *containes null*. while *_excludes_* are used
> later, NPE happens:
> {code:java}
> java.lang.NullPointerException
> at org.apache.hadoop.net.NodeBase.getPath(NodeBase.java:113)
> at
> org.apache.hadoop.net.NetworkTopology.countNumOfAvailableNodes(NetworkTopology.java:672)
> at
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:533)
> at
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:491)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.chooseDatanode(NamenodeWebHdfsMethods.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.redirectURI(NamenodeWebHdfsMethods.java:384)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.put(NamenodeWebHdfsMethods.java:652)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$2.run(NamenodeWebHdfsMethods.java:600)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$2.run(NamenodeWebHdfsMethods.java:597)
> at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:73)
> at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:30)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830)
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1086) Remove RaftClient from OM
[ https://issues.apache.org/jira/browse/HDDS-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772597#comment-16772597 ] Shashikant Banerjee commented on HDDS-1086: --- Thanks [~hanishakoneru]. In the case mentioned, we always let the ozone client aware of who the leader is in case a request gets redirected to a different OM Server so that next request it can send to a raft Client on OM1. If Ozone Client is only sending requests to OM2 and OM2 is not the leader, it will fail and the redirection still needs to happen to OM1 in such cases. > Remove RaftClient from OM > - > > Key: HDDS-1086 > URL: https://issues.apache.org/jira/browse/HDDS-1086 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: HA, OM >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HDDS-1086.001.patch > > > Currently we run RaftClient in OM which takes the incoming client requests > and submits it to the OM's Ratis server. This hop can be avoided if OM > submits the incoming client request directly to its Ratis server. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14299) error message when Operation mount point
[ https://issues.apache.org/jira/browse/HDFS-14299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hu xiaodong updated HDFS-14299: --- Attachment: HDFS-14299.001.patch > error message when Operation mount point > > > Key: HDFS-14299 > URL: https://issues.apache.org/jira/browse/HDFS-14299 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation >Affects Versions: 3.1.2 >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Minor > Attachments: .PNG, HDFS-14299.001.patch > > > when error occurred when operating mount point, the error message liks this: > !.PNG! > I think a separator should be included between "operation" and "Path" > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14299) error message when Operation mount point
hu xiaodong created HDFS-14299: -- Summary: error message when Operation mount point Key: HDFS-14299 URL: https://issues.apache.org/jira/browse/HDFS-14299 Project: Hadoop HDFS Issue Type: Bug Components: federation Affects Versions: 3.1.2 Reporter: hu xiaodong Attachments: .PNG when error occurred when operating mount point, the error message liks this: !.PNG! I think a separator should be included between "operation" and "Path" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14299) error message when Operation mount point
[ https://issues.apache.org/jira/browse/HDFS-14299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hu xiaodong reassigned HDFS-14299: -- Assignee: hu xiaodong > error message when Operation mount point > > > Key: HDFS-14299 > URL: https://issues.apache.org/jira/browse/HDFS-14299 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation >Affects Versions: 3.1.2 >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Minor > Attachments: .PNG > > > when error occurred when operating mount point, the error message liks this: > !.PNG! > I think a separator should be included between "operation" and "Path" > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1130) Make BenchMarkBlockManager multi-threaded
[ https://issues.apache.org/jira/browse/HDDS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772575#comment-16772575 ] Hudson commented on HDDS-1130: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16004 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16004/]) HDDS-1130. Make BenchMarkBlockManager multi-threaded. Contributed by (yqlin: rev 1d30fd94c6430492ce2f92883117eff56094eec0) * (edit) hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkBlockManager.java > Make BenchMarkBlockManager multi-threaded > - > > Key: HDDS-1130 > URL: https://issues.apache.org/jira/browse/HDDS-1130 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1130.001.patch, HDDS-1130.002.patch, > HDDS-1130.003.patch > > > Currently BenchMarkBlockManager is run by a single thread. We can make it > multi-threaded in order to have a better understanding of allocateBlock call > performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1130) Make BenchMarkBlockManager multi-threaded
[ https://issues.apache.org/jira/browse/HDDS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDDS-1130: Resolution: Fixed Target Version/s: 0.4.0 Status: Resolved (was: Patch Available) Committed this. Thanks [~ljain] for the contribution. > Make BenchMarkBlockManager multi-threaded > - > > Key: HDDS-1130 > URL: https://issues.apache.org/jira/browse/HDDS-1130 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1130.001.patch, HDDS-1130.002.patch, > HDDS-1130.003.patch > > > Currently BenchMarkBlockManager is run by a single thread. We can make it > multi-threaded in order to have a better understanding of allocateBlock call > performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1130) Make BenchMarkBlockManager multi-threaded
[ https://issues.apache.org/jira/browse/HDDS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772561#comment-16772561 ] Yiqun Lin commented on HDDS-1130: - LGTM, +1. Committing. > Make BenchMarkBlockManager multi-threaded > - > > Key: HDDS-1130 > URL: https://issues.apache.org/jira/browse/HDDS-1130 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1130.001.patch, HDDS-1130.002.patch, > HDDS-1130.003.patch > > > Currently BenchMarkBlockManager is run by a single thread. We can make it > multi-threaded in order to have a better understanding of allocateBlock call > performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14130) Make ZKFC ObserverNode aware
[
https://issues.apache.org/jira/browse/HDFS-14130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772554#comment-16772554
]
Konstantin Shvachko commented on HDFS-14130:
Was debugging this test some more. From Jenkins logs I can see it is failing
while waiting the STANDBY to transition to OBSERVER. {{waitForHAState()}} waits
for 15 seconds, then {{GenericTestUtils.waitFor()}} prints the thread dump. But
I cannot reproduce this locally. I assume there is some client-side retry logic
kicking in if we do state transitioning as {{DFSHAAdmin}} commands. So I
instead now use MiniDFSCluster commands in the test, which is equivalent to
{{DFSHAAdmin}}.
> Make ZKFC ObserverNode aware
>
>
> Key: HDFS-14130
> URL: https://issues.apache.org/jira/browse/HDFS-14130
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
>Affects Versions: HDFS-12943
>Reporter: Konstantin Shvachko
>Assignee: xiangheng
>Priority: Major
> Attachments: HDFS-14130-HDFS-12943.001.patch,
> HDFS-14130-HDFS-12943.003.patch, HDFS-14130-HDFS-12943.004.patch,
> HDFS-14130-HDFS-12943.005.patch, HDFS-14130-HDFS-12943.006.patch,
> HDFS-14130-HDFS-12943.007.patch, HDFS-14130.008.patch, HDFS-14130.009.patch
>
>
> Need to fix automatic failover with ZKFC. Currently it does not know about
> ObserverNodes trying to convert them to SBNs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14290) Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by DatanodeWebHdfsMethods
[ https://issues.apache.org/jira/browse/HDFS-14290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772547#comment-16772547 ] Lisheng Sun commented on HDFS-14290: Thanks [~jojochuang] for this issue. Later I will add a UT for reproducing the issue and corresponding comments. > Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by > DatanodeWebHdfsMethods > --- > > Key: HDFS-14290 > URL: https://issues.apache.org/jira/browse/HDFS-14290 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, webhdfs >Affects Versions: 2.7.0, 2.7.1 >Reporter: Lisheng Sun >Priority: Major > Attachments: HDFS-14290.000.patch, webhdfs show.png > > > The issue is there is no HttpRequestDecoder in InboundHandler of netty, > appear unexpected message type when read message. > > !webhdfs show.png! > DEBUG org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Proxy > failed. Cause: > com.xiaomi.infra.thirdparty.io.netty.handler.codec.EncoderException: > java.lang.IllegalStateException: unexpected message type: > PooledUnsafeDirectByteBuf > at > com.xiaomi.infra.thirdparty.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) > at > com.xiaomi.infra.thirdparty.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) > at > com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:304) > at > com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:137) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1051) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) > at > org.apache.hadoop.hdfs.server.datanode.web.SimpleHttpProxyHandler$Forwarder.channelRead(SimpleHttpProxyHandler.java:80) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:146) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) > at >
[jira] [Commented] (HDFS-14219) ConcurrentModificationException occurs in datanode occasionally
[ https://issues.apache.org/jira/browse/HDFS-14219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772535#comment-16772535 ] Tao Jie commented on HDFS-14219: Thank you [~vagarychen] for your comments. I've run failed unit tests on my local environment. All those tests passed. > ConcurrentModificationException occurs in datanode occasionally > --- > > Key: HDFS-14219 > URL: https://issues.apache.org/jira/browse/HDFS-14219 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.2 >Reporter: Tao Jie >Assignee: Tao Jie >Priority: Minor > Attachments: HDFS-14129-branch-2.8.001.patch, HDFS-14219.001.patch > > > ERROR occasionally occurs in datanode log: > ERROR BPServiceActor.java:792 - Exception in BPOfferService for Block pool > BP-1687106048-10.14.9.17-1535259994856 (Datanode Uuid > c17e635a-912f-4488-b4e8-93a58a27b5db) service to osscmh-9-21/10.14.9.21:8070 > java.util.ConcurrentModificationException: modification=62852685 != > iterModification = 62852684 > at > org.apache.hadoop.util.LightWeightGSet$SetIterator.ensureNext(LightWeightGSet.java:305) > at > org.apache.hadoop.util.LightWeightGSet$SetIterator.hasNext(LightWeightGSet.java:322) > at > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1872) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:349) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:648) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:790) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772528#comment-16772528
]
Íñigo Goiri commented on HDFS-14295:
Currently we have:
{code}
final String xferTargetsString = Arrays.asList(xferTargets).stream().map(n ->
n.toString()).collect(Collectors.joining(" "));
{code}
This is already transforming into a list.
Is java 8 smart enough to avoid extra compute?
The following looks cleaner if there is no extra overhead:
{code}
final String xferTargetsString = StringUtils.join(" ",
Arrays.asList(xferTargets);
{code}
Regarding the cast, can we make DataTransfer to be Callable?
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14130) Make ZKFC ObserverNode aware
[ https://issues.apache.org/jira/browse/HDFS-14130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-14130: --- Attachment: HDFS-14130.009.patch > Make ZKFC ObserverNode aware > > > Key: HDFS-14130 > URL: https://issues.apache.org/jira/browse/HDFS-14130 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: xiangheng >Priority: Major > Attachments: HDFS-14130-HDFS-12943.001.patch, > HDFS-14130-HDFS-12943.003.patch, HDFS-14130-HDFS-12943.004.patch, > HDFS-14130-HDFS-12943.005.patch, HDFS-14130-HDFS-12943.006.patch, > HDFS-14130-HDFS-12943.007.patch, HDFS-14130.008.patch, HDFS-14130.009.patch > > > Need to fix automatic failover with ZKFC. Currently it does not know about > ObserverNodes trying to convert them to SBNs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14081) hdfs dfsadmin -metasave metasave_test results NPE
[
https://issues.apache.org/jira/browse/HDFS-14081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772509#comment-16772509
]
Shweta commented on HDFS-14081:
---
Posted new patch 009 to address unit tests failures regarding metasave
functionality with HA.
Please review.
> hdfs dfsadmin -metasave metasave_test results NPE
> -
>
> Key: HDFS-14081
> URL: https://issues.apache.org/jira/browse/HDFS-14081
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1
>Reporter: Shweta
>Assignee: Shweta
>Priority: Major
> Attachments: HDFS-14081.001.patch, HDFS-14081.002.patch,
> HDFS-14081.003.patch, HDFS-14081.004.patch, HDFS-14081.005.patch,
> HDFS-14081.006.patch, HDFS-14081.007.patch, HDFS-14081.008.patch,
> HDFS-14081.009.patch
>
>
> Race condition is encountered while adding Block to
> postponedMisreplicatedBlocks which in turn tried to retrieve Block from
> BlockManager in which it may not be present.
> This happens in HA, metasave in first NN succeeded but failed in second NN,
> StackTrace showing NPE is as follows:
> {code}
> 2018-07-12 21:39:09,783 WARN org.apache.hadoop.ipc.Server: IPC Server handler
> 24 on 8020, call Call#1 Retry#0
> org.apache.hadoop.hdfs.protocol.ClientProtocol.metaSave from
> 172.26.9.163:602342018-07-12 21:39:09,783 WARN org.apache.hadoop.ipc.Server:
> IPC Server handler 24 on 8020, call Call#1 Retry#0
> org.apache.hadoop.hdfs.protocol.ClientProtocol.metaSave from
> 172.26.9.163:60234java.lang.NullPointerException at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseSourceDatanodes(BlockManager.java:2175)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.dumpBlockMeta(BlockManager.java:830)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.metaSave(BlockManager.java:762)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.metaSave(FSNamesystem.java:1782)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.metaSave(FSNamesystem.java:1766)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.metaSave(NameNodeRpcServer.java:1320)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.metaSave(ClientNamenodeProtocolServerSideTranslatorPB.java:928)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14081) hdfs dfsadmin -metasave metasave_test results NPE
[
https://issues.apache.org/jira/browse/HDFS-14081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shweta updated HDFS-14081:
--
Attachment: HDFS-14081.009.patch
> hdfs dfsadmin -metasave metasave_test results NPE
> -
>
> Key: HDFS-14081
> URL: https://issues.apache.org/jira/browse/HDFS-14081
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1
>Reporter: Shweta
>Assignee: Shweta
>Priority: Major
> Attachments: HDFS-14081.001.patch, HDFS-14081.002.patch,
> HDFS-14081.003.patch, HDFS-14081.004.patch, HDFS-14081.005.patch,
> HDFS-14081.006.patch, HDFS-14081.007.patch, HDFS-14081.008.patch,
> HDFS-14081.009.patch
>
>
> Race condition is encountered while adding Block to
> postponedMisreplicatedBlocks which in turn tried to retrieve Block from
> BlockManager in which it may not be present.
> This happens in HA, metasave in first NN succeeded but failed in second NN,
> StackTrace showing NPE is as follows:
> {code}
> 2018-07-12 21:39:09,783 WARN org.apache.hadoop.ipc.Server: IPC Server handler
> 24 on 8020, call Call#1 Retry#0
> org.apache.hadoop.hdfs.protocol.ClientProtocol.metaSave from
> 172.26.9.163:602342018-07-12 21:39:09,783 WARN org.apache.hadoop.ipc.Server:
> IPC Server handler 24 on 8020, call Call#1 Retry#0
> org.apache.hadoop.hdfs.protocol.ClientProtocol.metaSave from
> 172.26.9.163:60234java.lang.NullPointerException at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseSourceDatanodes(BlockManager.java:2175)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.dumpBlockMeta(BlockManager.java:830)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.metaSave(BlockManager.java:762)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.metaSave(FSNamesystem.java:1782)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.metaSave(FSNamesystem.java:1766)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.metaSave(NameNodeRpcServer.java:1320)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.metaSave(ClientNamenodeProtocolServerSideTranslatorPB.java:928)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14286) Logging stale datanode information
[ https://issues.apache.org/jira/browse/HDFS-14286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-14286: -- Resolution: Fixed Fix Version/s: 3.3.0 Status: Resolved (was: Patch Available) Thank You [~kpalanisamy] for the contribution. I have committed this to the trunk. > Logging stale datanode information > -- > > Key: HDFS-14286 > URL: https://issues.apache.org/jira/browse/HDFS-14286 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.1.2 >Reporter: Karthik Palanisamy >Assignee: Karthik Palanisamy >Priority: Trivial > Labels: supportability > Fix For: 3.3.0 > > Attachments: HDFS-14286-001.patch > > > Hard us to track stale Datanode information other than Namenode metrics but > this should be monitored. We need warning message at-least in the Namenode > log for debugging purpose. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14286) Logging stale datanode information
[ https://issues.apache.org/jira/browse/HDFS-14286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772492#comment-16772492 ] Bharat Viswanadham commented on HDFS-14286: --- +1 LGTM. Thank You [~kpalanisamy] for the fix. I will commit this shortly. > Logging stale datanode information > -- > > Key: HDFS-14286 > URL: https://issues.apache.org/jira/browse/HDFS-14286 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.1.2 >Reporter: Karthik Palanisamy >Assignee: Karthik Palanisamy >Priority: Trivial > Labels: supportability > Attachments: HDFS-14286-001.patch > > > Hard us to track stale Datanode information other than Namenode metrics but > this should be monitored. We need warning message at-least in the Namenode > log for debugging purpose. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1139) Fix findbugs issues caused by HDDS-1085.
[ https://issues.apache.org/jira/browse/HDDS-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772481#comment-16772481 ] Hudson commented on HDDS-1139: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16002 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16002/]) HDDS-1139 : Fix findbugs issues caused by HDDS-1085. Contributed by (bharat: rev e8d7e3b4e67e475f836b06180bd1f760d327f4bf) * (edit) hadoop-hdds/common/src/main/resources/ozone-default.xml * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStore.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBCheckpointManager.java * (edit) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java * (edit) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDbSnapshotServlet.java > Fix findbugs issues caused by HDDS-1085. > > > Key: HDDS-1139 > URL: https://issues.apache.org/jira/browse/HDDS-1139 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Attachments: HDDS-1139-000.patch > > Time Spent: 40m > Remaining Estimate: 0h > > The list of issues can be found here - > https://ci.anzix.net/job/ozone/106/findbugs/. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1085) Create an OM API to serve snapshots to Recon server
[ https://issues.apache.org/jira/browse/HDDS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772482#comment-16772482 ] Hudson commented on HDDS-1085: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16002 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16002/]) HDDS-1139 : Fix findbugs issues caused by HDDS-1085. Contributed by (bharat: rev e8d7e3b4e67e475f836b06180bd1f760d327f4bf) * (edit) hadoop-hdds/common/src/main/resources/ozone-default.xml * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStore.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBCheckpointManager.java * (edit) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java * (edit) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDbSnapshotServlet.java > Create an OM API to serve snapshots to Recon server > --- > > Key: HDDS-1085 > URL: https://issues.apache.org/jira/browse/HDDS-1085 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Siddharth Wagle >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Attachments: HDDS-1085-000.patch, HDDS-1085-001.patch, > HDDS-1085-002.patch, HDDS-1085-003.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > We need to add an API to OM so that we can serve snapshots from the OM server. > - The snapshot should be streamed to fsck server with the ability to > throttle network utilization (like TransferFsImage) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1139) Fix findbugs issues caused by HDDS-1085.
[ https://issues.apache.org/jira/browse/HDDS-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1139: - Resolution: Fixed Status: Resolved (was: Patch Available) Thank You [~avijayan] for the fix. I have committed this to trunk. > Fix findbugs issues caused by HDDS-1085. > > > Key: HDDS-1139 > URL: https://issues.apache.org/jira/browse/HDDS-1139 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Attachments: HDDS-1139-000.patch > > Time Spent: 40m > Remaining Estimate: 0h > > The list of issues can be found here - > https://ci.anzix.net/job/ozone/106/findbugs/. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1139) Fix findbugs issues caused by HDDS-1085.
[ https://issues.apache.org/jira/browse/HDDS-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1139: - Fix Version/s: 0.4.0 > Fix findbugs issues caused by HDDS-1085. > > > Key: HDDS-1139 > URL: https://issues.apache.org/jira/browse/HDDS-1139 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Attachments: HDDS-1139-000.patch > > Time Spent: 40m > Remaining Estimate: 0h > > The list of issues can be found here - > https://ci.anzix.net/job/ozone/106/findbugs/. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1139) Fix findbugs issues caused by HDDS-1085.
[ https://issues.apache.org/jira/browse/HDDS-1139?focusedWorklogId=200996=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-200996 ] ASF GitHub Bot logged work on HDDS-1139: Author: ASF GitHub Bot Created on: 20/Feb/19 00:32 Start Date: 20/Feb/19 00:32 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #504: HDDS-1139 : Fix findbugs issues caused by HDDS-1085. URL: https://github.com/apache/hadoop/pull/504 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 200996) Time Spent: 40m (was: 0.5h) > Fix findbugs issues caused by HDDS-1085. > > > Key: HDDS-1139 > URL: https://issues.apache.org/jira/browse/HDDS-1139 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Attachments: HDDS-1139-000.patch > > Time Spent: 40m > Remaining Estimate: 0h > > The list of issues can be found here - > https://ci.anzix.net/job/ozone/106/findbugs/. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14267) Add test_libhdfs_ops to libhdfs tests, mark libhdfs_read/write.c as examples
[
https://issues.apache.org/jira/browse/HDFS-14267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang reassigned HDFS-14267:
--
Assignee: Sahil Takiar
> Add test_libhdfs_ops to libhdfs tests, mark libhdfs_read/write.c as examples
>
>
> Key: HDFS-14267
> URL: https://issues.apache.org/jira/browse/HDFS-14267
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: libhdfs, native, test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HDFS-14267.001.patch, HDFS-14267.002.patch
>
>
> {{test_libhdfs_ops.c}} provides test coverage for basic operations against
> libhdfs, but currently has to be run manually (e.g. {{mvn install}} does not
> run these tests). The goal of this patch is to add {{test_libhdfs_ops.c}} to
> the list of tests that are automatically run for libhdfs.
> It looks like {{test_libhdfs_ops.c}} was used in conjunction with
> {{hadoop-hdfs-project/hadoop-hdfs/src/main/native/tests/test-libhdfs.sh}} to
> run some tests against a mini DFS cluster. Now that the
> {{NativeMiniDfsCluster}} exists, it makes more sense to use that rather than
> rely on an external bash script to start a mini DFS cluster.
> The {{libhdfs-tests}} directory (which contains {{test_libhdfs_ops.c}})
> contains two other files: {{test_libhdfs_read.c}} and
> {{test_libhdfs_write.c}}. At some point, these files might have been used in
> conjunction with {{test-libhdfs.sh}} to run some tests manually. However,
> they (1) largely overlap with the test coverage provided by
> {{test_libhdfs_ops.c}} and (2) are not designed to be run as unit tests. Thus
> I suggest we move these two files into a new folder called
> {{libhdfs-examples}} and use them to further document how users of libhdfs
> can use the API. We can move {{test-libhdfs.sh}} into the examples folder as
> well given that example files probably require the script to actually work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772457#comment-16772457
]
Erik Krogen commented on HDFS-14292:
Hi [~belugabehr], the reasoning makes sense to me but I'm curious if there is
any context that we are missing about why a new thread is used each time
previously, since a thread pool seems fairly obvious. Have you done any digging
in the history to see if anything is obvious?
Do you have any benchmarks to show that this strategy is better in any way?
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14111) hdfsOpenFile on HDFS causes unnecessary IO from file offset 0
[ https://issues.apache.org/jira/browse/HDFS-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772461#comment-16772461 ] Wei-Chiu Chuang commented on HDFS-14111: [~stakiar], assign the Jira to you. The patch makes sense to me. > hdfsOpenFile on HDFS causes unnecessary IO from file offset 0 > - > > Key: HDFS-14111 > URL: https://issues.apache.org/jira/browse/HDFS-14111 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client, libhdfs >Affects Versions: 3.2.0 >Reporter: Todd Lipcon >Assignee: Sahil Takiar >Priority: Major > > hdfsOpenFile() calls readDirect() with a 0-length argument in order to check > whether the underlying stream supports bytebuffer reads. With DFSInputStream, > the read(0) isn't short circuited, and results in the DFSClient opening a > block reader. In the case of a remote block, the block reader will actually > issue a read of the whole block, causing the datanode to perform unnecessary > IO and network transfers in order to fill up the client's TCP buffers. This > causes performance degradation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14111) hdfsOpenFile on HDFS causes unnecessary IO from file offset 0
[ https://issues.apache.org/jira/browse/HDFS-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reassigned HDFS-14111: -- Assignee: Sahil Takiar > hdfsOpenFile on HDFS causes unnecessary IO from file offset 0 > - > > Key: HDFS-14111 > URL: https://issues.apache.org/jira/browse/HDFS-14111 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client, libhdfs >Affects Versions: 3.2.0 >Reporter: Todd Lipcon >Assignee: Sahil Takiar >Priority: Major > > hdfsOpenFile() calls readDirect() with a 0-length argument in order to check > whether the underlying stream supports bytebuffer reads. With DFSInputStream, > the read(0) isn't short circuited, and results in the DFSClient opening a > block reader. In the case of a remote block, the block reader will actually > issue a read of the whole block, causing the datanode to perform unnecessary > IO and network transfers in order to fill up the client's TCP buffers. This > causes performance degradation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14297) Add cache for getContentSummary() result
[
https://issues.apache.org/jira/browse/HDFS-14297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772449#comment-16772449
]
Erik Krogen commented on HDFS-14297:
Hi [~Tao Jie], do the directories you are monitoring have quotas set on them?
If so, you can use the {{getQuotaUsage()}} API, which is a constant-time
operation.
> Add cache for getContentSummary() result
>
>
> Key: HDFS-14297
> URL: https://issues.apache.org/jira/browse/HDFS-14297
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Tao Jie
>Priority: Major
>
> In a large HDFS cluster, calling {{getContentSummary}} for a directory with
> large amount of files is very expensive. In a certain cluster with more than
> 100 million files, calling {{getContentSummary}} may take more than 10s and
> it will hold fsnamesystem lock for such a long time.
> In our cluster, there are several peripheral systems calling
> {{getContentSummary}} periodically to monitor the status of dirs. Actually we
> don't need the very accurate result in most cases. We could keep a cache for
> those contentSummary result in namenode, with which we could avoid repeated
> heavy request in a span. Also we should add more restrictions to this cache:
> 1,its size should be limited and it should be LRU, 2, only result of heavy
> request would be added to this cache, eg, rpctime over 1000ms.
> We may create a new RPC method or add a flag to the current method so that we
> will not modify the current behavior and we can have a choose of a accurate
> but expensive method or a fast but inaccurate method.
> Any thought?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1139) Fix findbugs issues caused by HDDS-1085.
[ https://issues.apache.org/jira/browse/HDDS-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-1139: Status: Patch Available (was: In Progress) > Fix findbugs issues caused by HDDS-1085. > > > Key: HDDS-1139 > URL: https://issues.apache.org/jira/browse/HDDS-1139 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Attachments: HDDS-1139-000.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > The list of issues can be found here - > https://ci.anzix.net/job/ozone/106/findbugs/. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1139) Fix findbugs issues caused by HDDS-1085.
[ https://issues.apache.org/jira/browse/HDDS-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-1139: Attachment: HDDS-1139-000.patch > Fix findbugs issues caused by HDDS-1085. > > > Key: HDDS-1139 > URL: https://issues.apache.org/jira/browse/HDDS-1139 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Attachments: HDDS-1139-000.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > The list of issues can be found here - > https://ci.anzix.net/job/ozone/106/findbugs/. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1139) Fix findbugs issues caused by HDDS-1085.
[ https://issues.apache.org/jira/browse/HDDS-1139?focusedWorklogId=200970=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-200970 ] ASF GitHub Bot logged work on HDDS-1139: Author: ASF GitHub Bot Created on: 19/Feb/19 23:24 Start Date: 19/Feb/19 23:24 Worklog Time Spent: 10m Work Description: avijayanhwx commented on issue #504: HDDS-1139 : Fix findbugs issues caused by HDDS-1085. URL: https://github.com/apache/hadoop/pull/504#issuecomment-465353620 All findbugs issues caused by HDDS-1085 have passed. The integration test TestOzoneConfigurationFields has also passed in the run. cc @elek @bharatviswa504 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 200970) Time Spent: 0.5h (was: 20m) > Fix findbugs issues caused by HDDS-1085. > > > Key: HDDS-1139 > URL: https://issues.apache.org/jira/browse/HDDS-1139 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The list of issues can be found here - > https://ci.anzix.net/job/ozone/106/findbugs/. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14293) Increase Default Size of dfs.stream-buffer-size
[
https://issues.apache.org/jira/browse/HDFS-14293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772434#comment-16772434
]
Shweta commented on HDFS-14293:
---
The value of 4096 for {{dfs.stream-buffer-size}} is not being used anywhere
meaning it doesn't seem to have any usages.
> Increase Default Size of dfs.stream-buffer-size
> ---
>
> Key: HDFS-14293
> URL: https://issues.apache.org/jira/browse/HDFS-14293
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Priority: Minor
>
> For many years (7+) now, the JDK has been using a default buffer size of
> [8192
> bytes|https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/io/BufferedInputStream.java#L53].
> Hadoop still defaults to a size half of that. The default is
> {{dfs.stream-buffer-size}} is 4096 bytes.
> https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
> Please increase default size to 8192.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14293) Increase Default Size of dfs.stream-buffer-size
[
https://issues.apache.org/jira/browse/HDFS-14293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772434#comment-16772434
]
Shweta edited comment on HDFS-14293 at 2/19/19 11:16 PM:
-
[~belugabehr], the value of 4096 for {{dfs.stream-buffer-size}} is not being
used anywhere meaning it doesn't seem to have any usages.
was (Author: shwetayakkali):
The value of 4096 for {{dfs.stream-buffer-size}} is not being used anywhere
meaning it doesn't seem to have any usages.
> Increase Default Size of dfs.stream-buffer-size
> ---
>
> Key: HDFS-14293
> URL: https://issues.apache.org/jira/browse/HDFS-14293
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Priority: Minor
>
> For many years (7+) now, the JDK has been using a default buffer size of
> [8192
> bytes|https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/io/BufferedInputStream.java#L53].
> Hadoop still defaults to a size half of that. The default is
> {{dfs.stream-buffer-size}} is 4096 bytes.
> https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
> Please increase default size to 8192.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1086) Remove RaftClient from OM
[ https://issues.apache.org/jira/browse/HDDS-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772427#comment-16772427 ] Hanisha Koneru commented on HDDS-1086: -- [~shashikant], even with Short Circuit read/ write in Ratis, we would still have the extra hop from a client talking to a non leader OM. Lets say OM1 is the Leader OM and we a have client sending requests to OM2. OM2's raft client would redirect the request to OM1 server. The client will not know that OM2 is not the leader and keep sending subsequent requests also to this follower node. This extra hop for every client request can be avoided if we submit requests directly to Ratis server and handle required features (Retries, Failover etc.) on OM client. > Remove RaftClient from OM > - > > Key: HDDS-1086 > URL: https://issues.apache.org/jira/browse/HDDS-1086 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: HA, OM >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HDDS-1086.001.patch > > > Currently we run RaftClient in OM which takes the incoming client requests > and submits it to the OM's Ratis server. This hop can be avoided if OM > submits the incoming client request directly to its Ratis server. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1060) Token: Add api to get OM certificate from SCM
[ https://issues.apache.org/jira/browse/HDDS-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-1060: - Attachment: HDDS-1060.04.patch > Token: Add api to get OM certificate from SCM > - > > Key: HDDS-1060 > URL: https://issues.apache.org/jira/browse/HDDS-1060 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: Blocker, Security > Fix For: 0.4.0 > > Attachments: HDDS-1060.00.patch, HDDS-1060.01.patch, > HDDS-1060.02.patch, HDDS-1060.03.patch, HDDS-1060.04.patch > > > Datanodes/OM need OM certificate to validate block tokens and delegation > tokens. > Add API for: > 1. getCertificate(String certSerialId): To get certificate from SCM based on > certificate serial id. > 2. getCACertificate(): To get CA certificate. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1109) Setup Failover Proxy Provider for client
[
https://issues.apache.org/jira/browse/HDDS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772401#comment-16772401
]
Hadoop QA commented on HDDS-1109:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
48s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 21s{color} | {color:orange} hadoop-ozone: The patch generated 4 new + 0
unchanged - 0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 49s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
27s{color} | {color:green} client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 54s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.TestOzoneConfigurationFields |
| | hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient |
| | hadoop.ozone.om.TestOzoneManagerHA |
| | hadoop.ozone.TestSecureOzoneCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1109 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-14290) Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by DatanodeWebHdfsMethods
[ https://issues.apache.org/jira/browse/HDFS-14290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772391#comment-16772391 ] Wei-Chiu Chuang commented on HDFS-14290: Hi Lisheng Sun, thanks a lot for reporting the issue. Would you also share how to reproduce? > Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by > DatanodeWebHdfsMethods > --- > > Key: HDFS-14290 > URL: https://issues.apache.org/jira/browse/HDFS-14290 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, webhdfs >Affects Versions: 2.7.0, 2.7.1 >Reporter: Lisheng Sun >Priority: Major > Attachments: HDFS-14290.000.patch, webhdfs show.png > > > The issue is there is no HttpRequestDecoder in InboundHandler of netty, > appear unexpected message type when read message. > > !webhdfs show.png! > DEBUG org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Proxy > failed. Cause: > com.xiaomi.infra.thirdparty.io.netty.handler.codec.EncoderException: > java.lang.IllegalStateException: unexpected message type: > PooledUnsafeDirectByteBuf > at > com.xiaomi.infra.thirdparty.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) > at > com.xiaomi.infra.thirdparty.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) > at > com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:304) > at > com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:137) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1051) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) > at > org.apache.hadoop.hdfs.server.datanode.web.SimpleHttpProxyHandler$Forwarder.channelRead(SimpleHttpProxyHandler.java:80) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:146) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) > at >
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772388#comment-16772388
]
BELUGA BEHR commented on HDFS-14295:
{{StringUtils#join}} is not the go-to option here because it requires a
{{Collection}} of Strings. That is not the case here.
A list of objects must be iterated to extract the required String first, then
joined. Nice to be able to do it in one-line.
In regards to the cast, I believe it's a necessary evil because of a mistake in
the Java API:
https://stackoverflow.com/questions/34233375/return-completablefuturevoid-or-completablefuture/34233790
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772387#comment-16772387
]
BELUGA BEHR commented on HDFS-14295:
{{StringUtils#join}} is not the go-to option here because it requires a
{{Collection}} of Strings. That is not the case here.
A list of objects must be iterated to extract the required String first, then
joined. Nice to be able to do it in one-line.
In regards to the cast, I believe it's a necessary evil because of a mistake in
the Java API:
https://stackoverflow.com/questions/34233375/return-completablefuturevoid-or-completablefuture/34233790
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1137) Evaluate and move over OM RocskDB checkpoint request API to RPC.
[ https://issues.apache.org/jira/browse/HDDS-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-1137: Description: As per the discussion on HDDS-1085, we have to evaluate * Moving over rocksdb checkpoint request to be perhaps invoked via RPC to create snapshot and HTTP to be read an existing snapshot instead of doing all that operation in a single blocking call. cc [~anu] * Add functional testing to cover end to end test of checkpoint download/streaming capability. * Evaluate the need to tar/untar the checkpoint artifact generated by RocksDB. was: As per the discussion on HDDS-1085, we have to evaluate moving over rocksdb checkpoint request to be perhaps invoked via RPC to create snapshot and HTTP to be read an existing snapshot instead of doing all that operation in a single blocking call. cc [~anu] Add functional testing to cover end to end test of checkpoint download/streaming capability. > Evaluate and move over OM RocskDB checkpoint request API to RPC. > > > Key: HDDS-1137 > URL: https://issues.apache.org/jira/browse/HDDS-1137 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > > As per the discussion on HDDS-1085, we have to evaluate > * Moving over rocksdb checkpoint request to be perhaps invoked via RPC to > create snapshot and HTTP to be read an existing snapshot instead of doing all > that operation in a single blocking call. cc [~anu] > * Add functional testing to cover end to end test of checkpoint > download/streaming capability. > * Evaluate the need to tar/untar the checkpoint artifact generated by > RocksDB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1137) Evaluate and move over OM RocskDB checkpoint request API to RPC.
[ https://issues.apache.org/jira/browse/HDDS-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-1137: Description: As per the discussion on HDDS-1085, we have to evaluate moving over rocksdb checkpoint request to be perhaps invoked via RPC to create snapshot and HTTP to be read an existing snapshot instead of doing all that operation in a single blocking call. cc [~anu] Add functional testing to cover end to end test of checkpoint download/streaming capability. was:As per the discussion on HDDS-1085, we have to evaluate moving over rocksdb checkpoint request to be perhaps invoked via RPC to create snapshot and HTTP to be read an existing snapshot instead of doing all that operation in a single blocking call. cc [~anu] > Evaluate and move over OM RocskDB checkpoint request API to RPC. > > > Key: HDDS-1137 > URL: https://issues.apache.org/jira/browse/HDDS-1137 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > > As per the discussion on HDDS-1085, we have to evaluate moving over rocksdb > checkpoint request to be perhaps invoked via RPC to create snapshot and HTTP > to be read an existing snapshot instead of doing all that operation in a > single blocking call. cc [~anu] > Add functional testing to cover end to end test of checkpoint > download/streaming capability. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1141) Update DBCheckpointSnapshot to DBCheckpoint
[ https://issues.apache.org/jira/browse/HDDS-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1141: - Attachment: HDDS-1141.01.patch > Update DBCheckpointSnapshot to DBCheckpoint > --- > > Key: HDDS-1141 > URL: https://issues.apache.org/jira/browse/HDDS-1141 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Minor > Fix For: 0.4.0 > > Attachments: HDDS-1141.00.patch, HDDS-1141.01.patch > > > Instead of DBCheckpointSnapshot to DBCheckpoint, as internally we used > RocksDBCheckpoint. It is little confusing to have both checkpoint and > snapshot. And update similarly in other places. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1141) Update DBCheckpointSnapshot to DBCheckpoint
[ https://issues.apache.org/jira/browse/HDDS-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772354#comment-16772354 ] Bharat Viswanadham commented on HDDS-1141: -- Thank You [~avijayan] for review. I have addressed review comment in patch v01. > Update DBCheckpointSnapshot to DBCheckpoint > --- > > Key: HDDS-1141 > URL: https://issues.apache.org/jira/browse/HDDS-1141 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Minor > Fix For: 0.4.0 > > Attachments: HDDS-1141.00.patch, HDDS-1141.01.patch > > > Instead of DBCheckpointSnapshot to DBCheckpoint, as internally we used > RocksDBCheckpoint. It is little confusing to have both checkpoint and > snapshot. And update similarly in other places. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1109) Setup Failover Proxy Provider for client
[ https://issues.apache.org/jira/browse/HDDS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-1109: - Attachment: HDDS-1109.002.patch > Setup Failover Proxy Provider for client > > > Key: HDDS-1109 > URL: https://issues.apache.org/jira/browse/HDDS-1109 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HDDS-1109.001.patch, HDDS-1109.002.patch > > > If a client sends a request to a OM follower, it will get a > NotLeaderException. The client should then keep trying the request on other > OMs till it finds the leader OM. Client should cache the information about > current OM leader node. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1109) Setup Failover Proxy Provider for client
[ https://issues.apache.org/jira/browse/HDDS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772341#comment-16772341 ] Hanisha Koneru commented on HDDS-1109: -- Thank you for the review, [~linyiqun]. Added a unit test in patch v02. > Setup Failover Proxy Provider for client > > > Key: HDDS-1109 > URL: https://issues.apache.org/jira/browse/HDDS-1109 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HDDS-1109.001.patch, HDDS-1109.002.patch > > > If a client sends a request to a OM follower, it will get a > NotLeaderException. The client should then keep trying the request on other > OMs till it finds the leader OM. Client should cache the information about > current OM leader node. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1109) Setup Failover Proxy Provider for client
[ https://issues.apache.org/jira/browse/HDDS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-1109: - Status: Patch Available (was: Open) > Setup Failover Proxy Provider for client > > > Key: HDDS-1109 > URL: https://issues.apache.org/jira/browse/HDDS-1109 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HDDS-1109.001.patch, HDDS-1109.002.patch > > > If a client sends a request to a OM follower, it will get a > NotLeaderException. The client should then keep trying the request on other > OMs till it finds the leader OM. Client should cache the information about > current OM leader node. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1141) Update DBCheckpointSnapshot to DBCheckpoint
[
https://issues.apache.org/jira/browse/HDDS-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772335#comment-16772335
]
Aravindan Vijayan commented on HDDS-1141:
-
The following keys have to be probably changed as well
{code}
public static final String OZONE_DB_SNAPSHOT_TRANSFER_RATE_KEY =
"ozone.manager.db.snapshot.transfer.bandwidthPerSec";
public static final long OZONE_DB_SNAPSHOT_TRANSFER_RATE_DEFAULT =
0; //no throttling
{code}
> Update DBCheckpointSnapshot to DBCheckpoint
> ---
>
> Key: HDDS-1141
> URL: https://issues.apache.org/jira/browse/HDDS-1141
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Minor
> Fix For: 0.4.0
>
> Attachments: HDDS-1141.00.patch
>
>
> Instead of DBCheckpointSnapshot to DBCheckpoint, as internally we used
> RocksDBCheckpoint. It is little confusing to have both checkpoint and
> snapshot. And update similarly in other places.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14052) RBF: Use Router keytab for WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772327#comment-16772327
]
Íñigo Goiri commented on HDFS-14052:
[~brahmareddy], I think the current unit tests that this starts.
Is there anything else you want to check?
> RBF: Use Router keytab for WebHDFS
> --
>
> Key: HDFS-14052
> URL: https://issues.apache.org/jira/browse/HDFS-14052
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14052-HDFS-13891.0.patch
>
>
> When the RouterHttpServer starts it does:
> {code}
> NameNodeHttpServer.initWebHdfs(conf, httpAddress.getHostName(),
> httpServer,
> RouterWebHdfsMethods.class.getPackage().getName());
> {code}
> This function is in the NN and is pretty generic.
> However, it then calls to NameNodeHttpServer#getAuthFilterParams, which does:
> {code}
> String httpKeytab = conf.get(DFSUtil.getSpnegoKeytabKey(conf,
> DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY));
> {code}
> In most cases, the regular web keytab will kick in, but we should make this a
> parameter and load the Router one just in case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDDS-1139) Fix findbugs issues caused by HDDS-1085.
[ https://issues.apache.org/jira/browse/HDDS-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDDS-1139 started by Aravindan Vijayan. --- > Fix findbugs issues caused by HDDS-1085. > > > Key: HDDS-1139 > URL: https://issues.apache.org/jira/browse/HDDS-1139 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The list of issues can be found here - > https://ci.anzix.net/job/ozone/106/findbugs/. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1139) Fix findbugs issues caused by HDDS-1085.
[ https://issues.apache.org/jira/browse/HDDS-1139?focusedWorklogId=200863=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-200863 ] ASF GitHub Bot logged work on HDDS-1139: Author: ASF GitHub Bot Created on: 19/Feb/19 20:35 Start Date: 19/Feb/19 20:35 Worklog Time Spent: 10m Work Description: avijayanhwx commented on pull request #503: HDDS-1139 : Fix findbugs issues in HDDS-1085. URL: https://github.com/apache/hadoop/pull/503 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 200863) Time Spent: 10m Remaining Estimate: 0h > Fix findbugs issues caused by HDDS-1085. > > > Key: HDDS-1139 > URL: https://issues.apache.org/jira/browse/HDDS-1139 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The list of issues can be found here - > https://ci.anzix.net/job/ozone/106/findbugs/. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14298) Improve log messages of ECTopologyVerifier
[ https://issues.apache.org/jira/browse/HDFS-14298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772314#comment-16772314 ] Shweta commented on HDFS-14298: --- Thanks for the patch [~knanasi]. Looks good to me. +1 (Non-binding) > Improve log messages of ECTopologyVerifier > -- > > Key: HDFS-14298 > URL: https://issues.apache.org/jira/browse/HDFS-14298 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Minor > Attachments: HDFS-14298.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1139) Fix findbugs issues caused by HDDS-1085.
[ https://issues.apache.org/jira/browse/HDDS-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1139: - Labels: pull-request-available (was: ) > Fix findbugs issues caused by HDDS-1085. > > > Key: HDDS-1139 > URL: https://issues.apache.org/jira/browse/HDDS-1139 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > > The list of issues can be found here - > https://ci.anzix.net/job/ozone/106/findbugs/. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1139) Fix findbugs issues caused by HDDS-1085.
[ https://issues.apache.org/jira/browse/HDDS-1139?focusedWorklogId=200864=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-200864 ] ASF GitHub Bot logged work on HDDS-1139: Author: ASF GitHub Bot Created on: 19/Feb/19 20:35 Start Date: 19/Feb/19 20:35 Worklog Time Spent: 10m Work Description: avijayanhwx commented on pull request #504: HDDS-1139 : Fix findbugs issues caused by HDDS-1085. URL: https://github.com/apache/hadoop/pull/504 Fixing findbugs issues caused by HDDS-1085. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 200864) Time Spent: 20m (was: 10m) > Fix findbugs issues caused by HDDS-1085. > > > Key: HDDS-1139 > URL: https://issues.apache.org/jira/browse/HDDS-1139 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The list of issues can be found here - > https://ci.anzix.net/job/ozone/106/findbugs/. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14188) Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a parameter
[
https://issues.apache.org/jira/browse/HDFS-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772311#comment-16772311
]
Hudson commented on HDFS-14188:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16000 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16000/])
HDFS-14188. Make hdfs ec -verifyClusterSetup command accept an erasure
(weichiu: rev 14282e311be6ffcaddd2f74fa8e67c4e98a32291)
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestECAdmin.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/ECAdmin.java
> Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a
> parameter
> ---
>
> Key: HDFS-14188
> URL: https://issues.apache.org/jira/browse/HDFS-14188
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.1.1
>Reporter: Kitti Nanasi
>Assignee: Kitti Nanasi
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14188.001.patch, HDFS-14188.002.patch,
> HDFS-14188.003.patch, HDFS-14188.004.patch, HDFS-14188.005.patch
>
>
> hdfs ec -verifyClusterSetup command verifies if there are enough data nodes
> and racks for the enabled erasure coding policies
> I think it would be beneficial if it could accept an erasure coding policy as
> a parameter optionally. For example the following command would run the
> verify for only the RS-6-3-1024k policy.
> {code:java}
> hdfs ec -verifyClusterSetup -policy RS-6-3-1024k
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1141) Update DBCheckpointSnapshot to DBCheckpoint
[ https://issues.apache.org/jira/browse/HDDS-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1141: - Attachment: HDDS-1141.00.patch > Update DBCheckpointSnapshot to DBCheckpoint > --- > > Key: HDDS-1141 > URL: https://issues.apache.org/jira/browse/HDDS-1141 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Minor > Attachments: HDDS-1141.00.patch > > > Instead of DBCheckpointSnapshot to DBCheckpoint, as internally we used > RocksDBCheckpoint. It is little confusing to have both checkpoint and > snapshot. And update similarly in other places. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1141) Update DBCheckpointSnapshot to DBCheckpoint
[ https://issues.apache.org/jira/browse/HDDS-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1141: - Fix Version/s: 0.4.0 > Update DBCheckpointSnapshot to DBCheckpoint > --- > > Key: HDDS-1141 > URL: https://issues.apache.org/jira/browse/HDDS-1141 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Minor > Fix For: 0.4.0 > > Attachments: HDDS-1141.00.patch > > > Instead of DBCheckpointSnapshot to DBCheckpoint, as internally we used > RocksDBCheckpoint. It is little confusing to have both checkpoint and > snapshot. And update similarly in other places. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1141) Update DBCheckpointSnapshot to DBCheckpoint
[ https://issues.apache.org/jira/browse/HDDS-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1141: - Status: Patch Available (was: Open) > Update DBCheckpointSnapshot to DBCheckpoint > --- > > Key: HDDS-1141 > URL: https://issues.apache.org/jira/browse/HDDS-1141 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Minor > Attachments: HDDS-1141.00.patch > > > Instead of DBCheckpointSnapshot to DBCheckpoint, as internally we used > RocksDBCheckpoint. It is little confusing to have both checkpoint and > snapshot. And update similarly in other places. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1141) Update DBCheckpointSnapshot to DBCheckpoint
[ https://issues.apache.org/jira/browse/HDDS-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1141: - Description: Instead of DBCheckpointSnapshot to DBCheckpoint, as internally we used RocksDBCheckpoint. It is little confusing to have both checkpoint and snapshot. And update similarly in other places. (was: Instead of DBCheckpointSnapshot to DBCheckpoint, as internally we used RocksDBCheckpoint. It is little confusing to have both checkpoint and snapshot.) > Update DBCheckpointSnapshot to DBCheckpoint > --- > > Key: HDDS-1141 > URL: https://issues.apache.org/jira/browse/HDDS-1141 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Minor > > Instead of DBCheckpointSnapshot to DBCheckpoint, as internally we used > RocksDBCheckpoint. It is little confusing to have both checkpoint and > snapshot. And update similarly in other places. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1141) Update DBCheckpointSnapshot to DBCheckpoint
[ https://issues.apache.org/jira/browse/HDDS-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1141: - Issue Type: Sub-task (was: Bug) Parent: HDDS-1084 > Update DBCheckpointSnapshot to DBCheckpoint > --- > > Key: HDDS-1141 > URL: https://issues.apache.org/jira/browse/HDDS-1141 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Minor > > Instead of DBCheckpointSnapshot to DBCheckpoint, as internally we used > RocksDBCheckpoint. It is little confusing to have both checkpoint and > snapshot. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1141) Update DBCheckpointSnapshot to DBCheckpoint
Bharat Viswanadham created HDDS-1141: Summary: Update DBCheckpointSnapshot to DBCheckpoint Key: HDDS-1141 URL: https://issues.apache.org/jira/browse/HDDS-1141 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham Instead of DBCheckpointSnapshot to DBCheckpoint, as internally we used RocksDBCheckpoint. It is little confusing to have both checkpoint and snapshot. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1140) TestSCMChillModeManager is failing with NullPointerException
Mukul Kumar Singh created HDDS-1140:
---
Summary: TestSCMChillModeManager is failing with
NullPointerException
Key: HDDS-1140
URL: https://issues.apache.org/jira/browse/HDDS-1140
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: SCM
Affects Versions: 0.4.0
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh
TestSCMChillModeManager is failing with the following exception
{code}
[ERROR]
testDisableChillMode(org.apache.hadoop.hdds.scm.chillmode.TestSCMChillModeManager)
Time elapsed: 0.012 s <<< ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.utils.Scheduler.scheduleWithFixedDelay(Scheduler.java:78)
at
org.apache.hadoop.hdds.scm.pipeline.RatisPipelineUtils.scheduleFixedIntervalPipelineCreator(RatisPipelineUtils.java:211)
at
org.apache.hadoop.hdds.scm.chillmode.SCMChillModeManager.exitChillMode(SCMChillModeManager.java:137)
at
org.apache.hadoop.hdds.scm.chillmode.SCMChillModeManager.(SCMChillModeManager.java:93)
at
org.apache.hadoop.hdds.scm.chillmode.TestSCMChillModeManager.testDisableChillMode(TestSCMChillModeManager.java:134)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14052) RBF: Use Router keytab for WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772291#comment-16772291
]
Hadoop QA commented on HDFS-14052:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
30s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 23m
6s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14052 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947269/HDFS-14052-HDFS-13891.0.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 05d32e0558a8 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / f94b6e3 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26261/testReport/ |
| Max. process+thread count | 1036 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26261/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically
[jira] [Created] (HDDS-1139) Fix findbugs issues caused bu HDDS-1085.
Aravindan Vijayan created HDDS-1139: --- Summary: Fix findbugs issues caused bu HDDS-1085. Key: HDDS-1139 URL: https://issues.apache.org/jira/browse/HDDS-1139 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Aravindan Vijayan Assignee: Aravindan Vijayan The list of issues can be found here - https://ci.anzix.net/job/ozone/106/findbugs/. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1139) Fix findbugs issues caused by HDDS-1085.
[ https://issues.apache.org/jira/browse/HDDS-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-1139: Summary: Fix findbugs issues caused by HDDS-1085. (was: Fix findbugs issues caused bu HDDS-1085.) > Fix findbugs issues caused by HDDS-1085. > > > Key: HDDS-1139 > URL: https://issues.apache.org/jira/browse/HDDS-1139 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > > The list of issues can be found here - > https://ci.anzix.net/job/ozone/106/findbugs/. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14188) Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a parameter
[
https://issues.apache.org/jira/browse/HDFS-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14188:
---
Resolution: Fixed
Fix Version/s: 3.3.0
Status: Resolved (was: Patch Available)
Pushed to trunk. Thanks [~knanasi] for the contribution, [~shwetayakkali] and
[~ayushtkn] for review.
> Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a
> parameter
> ---
>
> Key: HDFS-14188
> URL: https://issues.apache.org/jira/browse/HDFS-14188
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.1.1
>Reporter: Kitti Nanasi
>Assignee: Kitti Nanasi
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14188.001.patch, HDFS-14188.002.patch,
> HDFS-14188.003.patch, HDFS-14188.004.patch, HDFS-14188.005.patch
>
>
> hdfs ec -verifyClusterSetup command verifies if there are enough data nodes
> and racks for the enabled erasure coding policies
> I think it would be beneficial if it could accept an erasure coding policy as
> a parameter optionally. For example the following command would run the
> verify for only the RS-6-3-1024k policy.
> {code:java}
> hdfs ec -verifyClusterSetup -policy RS-6-3-1024k
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1135) Ozone jars are missing in the Ozone Snapshot tar
[
https://issues.apache.org/jira/browse/HDDS-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772278#comment-16772278
]
Hadoop QA commented on HDDS-1135:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
33m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} | {color:red} dist in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
16s{color} | {color:red} dist in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 16s{color}
| {color:red} dist in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
18s{color} | {color:red} dist in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 16s{color}
| {color:red} dist in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1135 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959306/HDDS-1135.00.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux fda9e83e3752 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / cf1a66d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDDS-Build/2311/artifact/out/patch-mvninstall-hadoop-ozone_dist.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDDS-Build/2311/artifact/out/patch-compile-hadoop-ozone_dist.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDDS-Build/2311/artifact/out/patch-compile-hadoop-ozone_dist.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDDS-Build/2311/artifact/out/patch-mvnsite-hadoop-ozone_dist.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/2311/artifact/out/patch-unit-hadoop-ozone_dist.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/2311/testReport/ |
| Max. process+thread count | 446 (vs. ulimit of 1) |
| modules | C: hadoop-ozone/dist U: hadoop-ozone/dist |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/2311/console |
| Powered by | Apache Yetus
[jira] [Updated] (HDDS-1138) OzoneManager should return the pipeline info of the allocated block along with block info
[ https://issues.apache.org/jira/browse/HDDS-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-1138: Attachment: HDDS-1138.001.patch > OzoneManager should return the pipeline info of the allocated block along > with block info > - > > Key: HDDS-1138 > URL: https://issues.apache.org/jira/browse/HDDS-1138 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client, Ozone Manager >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Attachments: HDDS-1138.001.patch > > > Currently, while a block is allocated from OM, the request is forwarded to > SCM. However, even though the pipeline information is present with the OM for > block allocation, this information is passed through to the client. > This optimization will help in reducing the number of hops for the client by > reducing 1 RPC round trip for each block allocated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1085) Create an OM API to serve snapshots to Recon server
[ https://issues.apache.org/jira/browse/HDDS-1085?focusedWorklogId=200835=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-200835 ] ASF GitHub Bot logged work on HDDS-1085: Author: ASF GitHub Bot Created on: 19/Feb/19 19:39 Start Date: 19/Feb/19 19:39 Worklog Time Spent: 10m Work Description: avijayanhwx commented on pull request #503: Fix findbugs issues in HDDS-1085. URL: https://github.com/apache/hadoop/pull/503 Fixing findbugs issues in HDDS-1085. If all the issues are solved, I will create a JIRA and attach the patch. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 200835) Time Spent: 0.5h (was: 20m) > Create an OM API to serve snapshots to Recon server > --- > > Key: HDDS-1085 > URL: https://issues.apache.org/jira/browse/HDDS-1085 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Siddharth Wagle >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Attachments: HDDS-1085-000.patch, HDDS-1085-001.patch, > HDDS-1085-002.patch, HDDS-1085-003.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > We need to add an API to OM so that we can serve snapshots from the OM server. > - The snapshot should be streamed to fsck server with the ability to > throttle network utilization (like TransferFsImage) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14219) ConcurrentModificationException occurs in datanode occasionally
[ https://issues.apache.org/jira/browse/HDFS-14219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772269#comment-16772269 ] Chen Liang commented on HDFS-14219: --- Thanks for reporting [~Tao Jie]! I think this discrepancy was introduced in HDFS-9781 though. The change looks good to me. Can you verify the failed unit tests? > ConcurrentModificationException occurs in datanode occasionally > --- > > Key: HDFS-14219 > URL: https://issues.apache.org/jira/browse/HDFS-14219 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.2 >Reporter: Tao Jie >Assignee: Tao Jie >Priority: Minor > Attachments: HDFS-14129-branch-2.8.001.patch, HDFS-14219.001.patch > > > ERROR occasionally occurs in datanode log: > ERROR BPServiceActor.java:792 - Exception in BPOfferService for Block pool > BP-1687106048-10.14.9.17-1535259994856 (Datanode Uuid > c17e635a-912f-4488-b4e8-93a58a27b5db) service to osscmh-9-21/10.14.9.21:8070 > java.util.ConcurrentModificationException: modification=62852685 != > iterModification = 62852684 > at > org.apache.hadoop.util.LightWeightGSet$SetIterator.ensureNext(LightWeightGSet.java:305) > at > org.apache.hadoop.util.LightWeightGSet$SetIterator.hasNext(LightWeightGSet.java:322) > at > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1872) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:349) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:648) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:790) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1121) Key read failure when data is written parallel in to Ozone
[
https://issues.apache.org/jira/browse/HDDS-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772270#comment-16772270
]
Hudson commented on HDDS-1121:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15997 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15997/])
HDDS-1121. Key read failure when data is written parallel in to Ozone.
(aengineer: rev 02d04bd1073c2ae89fd0b208eba239a469559f90)
* (edit)
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java
* (edit)
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockOutputStreamEntry.java
* (edit)
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java
* (edit)
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
* (edit)
hadoop-ozone/objectstore-service/src/main/java/org/apache/hadoop/ozone/web/storage/DistributedStorageHandler.java
* (edit)
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/Checksum.java
* (edit)
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
> Key read failure when data is written parallel in to Ozone
> --
>
> Key: HDDS-1121
> URL: https://issues.apache.org/jira/browse/HDDS-1121
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1121.00.patch, HDDS-1121.01.patch,
> HDDS-1121.02.patch
>
>
> When hive is run with multiple threads for data ingestion to ozone. After
> ingestion is done, during read we see this below error.
> This issue is found during hive testing, and found by [~t3rmin4t0r]
> {code:java}
> caused by: org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum
> mismatch at index 0
> at
> org.apache.hadoop.ozone.common.ChecksumData.verifyChecksumDataMatches(ChecksumData.java:143)
> at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:239)
> at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:217)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:227)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.seek(BlockInputStream.java:259)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.seek(KeyInputStream.java:249)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream.seek(KeyInputStream.java:180)
> at
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.seek(OzoneFSInputStream.java:62)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:82)
> at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
> at
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:555)
> at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370)
> at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:61)
> at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:105)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:1647)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1533)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2700(OrcInputFormat.java:1329)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1513)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1510)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1510)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1329)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1085) Create an OM API to serve snapshots to Recon server
[ https://issues.apache.org/jira/browse/HDDS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772249#comment-16772249 ] Aravindan Vijayan commented on HDDS-1085: - Thanks [~anu]. Created HDDS-1136 and HDDS-1137 to track the work items. > Create an OM API to serve snapshots to Recon server > --- > > Key: HDDS-1085 > URL: https://issues.apache.org/jira/browse/HDDS-1085 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Siddharth Wagle >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Attachments: HDDS-1085-000.patch, HDDS-1085-001.patch, > HDDS-1085-002.patch, HDDS-1085-003.patch > > Time Spent: 10m > Remaining Estimate: 0h > > We need to add an API to OM so that we can serve snapshots from the OM server. > - The snapshot should be streamed to fsck server with the ability to > throttle network utilization (like TransferFsImage) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1085) Create an OM API to serve snapshots to Recon server
[ https://issues.apache.org/jira/browse/HDDS-1085?focusedWorklogId=200826=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-200826 ] ASF GitHub Bot logged work on HDDS-1085: Author: ASF GitHub Bot Created on: 19/Feb/19 19:22 Start Date: 19/Feb/19 19:22 Worklog Time Spent: 10m Work Description: avijayanhwx commented on pull request #494: HDDS-1085 : Create an OM API to serve snapshots to Recon server. URL: https://github.com/apache/hadoop/pull/494 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 200826) Time Spent: 20m (was: 10m) > Create an OM API to serve snapshots to Recon server > --- > > Key: HDDS-1085 > URL: https://issues.apache.org/jira/browse/HDDS-1085 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Siddharth Wagle >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Attachments: HDDS-1085-000.patch, HDDS-1085-001.patch, > HDDS-1085-002.patch, HDDS-1085-003.patch > > Time Spent: 20m > Remaining Estimate: 0h > > We need to add an API to OM so that we can serve snapshots from the OM server. > - The snapshot should be streamed to fsck server with the ability to > throttle network utilization (like TransferFsImage) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1137) Evaluate and move over OM RocskDB checkpoint request API to RPC.
Aravindan Vijayan created HDDS-1137: --- Summary: Evaluate and move over OM RocskDB checkpoint request API to RPC. Key: HDDS-1137 URL: https://issues.apache.org/jira/browse/HDDS-1137 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Aravindan Vijayan Assignee: Aravindan Vijayan As per the discussion on HDDS-1085, we have to evaluate moving over rocksdb checkpoint request to be perhaps invoked via RPC to create snapshot and HTTP to be read an existing snapshot instead of doing all that operation in a single blocking call. cc [~anu] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1138) OzoneManager should return the pipeline info of the allocated block along with block info
Mukul Kumar Singh created HDDS-1138: --- Summary: OzoneManager should return the pipeline info of the allocated block along with block info Key: HDDS-1138 URL: https://issues.apache.org/jira/browse/HDDS-1138 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client, Ozone Manager Affects Versions: 0.4.0 Reporter: Mukul Kumar Singh Assignee: Mukul Kumar Singh Currently, while a block is allocated from OM, the request is forwarded to SCM. However, even though the pipeline information is present with the OM for block allocation, this information is passed through to the client. This optimization will help in reducing the number of hops for the client by reducing 1 RPC round trip for each block allocated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1121) Key read failure when data is written parallel in to Ozone
[
https://issues.apache.org/jira/browse/HDDS-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDDS-1121:
---
Resolution: Fixed
Fix Version/s: 0.4.0
Status: Resolved (was: Patch Available)
[~linyiqun], [~shashikant] Thanks for the reviews. [~bharatviswa] Thanks for
fixiing this issue. [~gopalv] thanks for reporting this issue. I have committed
this to the trunk after verifying, checkstyle, findbugs etc. on the local
machine. ASF infra seems to have some issues, since it has been 2 days since
the patch is posted and jenkins have not picked up. I have verified these
things manually.
> Key read failure when data is written parallel in to Ozone
> --
>
> Key: HDDS-1121
> URL: https://issues.apache.org/jira/browse/HDDS-1121
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1121.00.patch, HDDS-1121.01.patch,
> HDDS-1121.02.patch
>
>
> When hive is run with multiple threads for data ingestion to ozone. After
> ingestion is done, during read we see this below error.
> This issue is found during hive testing, and found by [~t3rmin4t0r]
> {code:java}
> caused by: org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum
> mismatch at index 0
> at
> org.apache.hadoop.ozone.common.ChecksumData.verifyChecksumDataMatches(ChecksumData.java:143)
> at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:239)
> at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:217)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:227)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.seek(BlockInputStream.java:259)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.seek(KeyInputStream.java:249)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream.seek(KeyInputStream.java:180)
> at
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.seek(OzoneFSInputStream.java:62)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:82)
> at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
> at
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:555)
> at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370)
> at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:61)
> at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:105)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:1647)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1533)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2700(OrcInputFormat.java:1329)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1513)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1510)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1510)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1329)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1136) Add metric counters to capture the RocksDB checkpointing statistics.
Aravindan Vijayan created HDDS-1136: --- Summary: Add metric counters to capture the RocksDB checkpointing statistics. Key: HDDS-1136 URL: https://issues.apache.org/jira/browse/HDDS-1136 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Aravindan Vijayan Assignee: Aravindan Vijayan As per the discussion with [~anu] on HDDS-1085, this JIRA tracks the effort to add metric counters to capture ROcksDB checkpointing performance. >From [~anu]'s comments, it might be interesting to have 3 counters – or a map >of counters. * How much time are we taking for each CheckPoint * How much time are we taking for each Tar operation – along with sizes * How much time are we taking for the transfer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1095) OzoneManager#openKey should do multiple block allocations in a single SCM rpc call
[ https://issues.apache.org/jira/browse/HDDS-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772238#comment-16772238 ] Bharat Viswanadham commented on HDDS-1095: -- Thank You [~msingh] for this perf improvement fix. *Few comments:* # Line 417 can we move before calling allocateBlocks, and remove the preAllocate and allocateSize? In this way even if the key length provided is zero, we allocate at least a block, as we are setting size as scmBlockSize and storing as keyInfo. (As even now we are doing min(Preallocate, length), so with above mentioned way, we can remove this logic. # And also code comments in allocate locks and in openKey Line 406 needs to be updated. # Can we add a javadoc for newly added method allocateBlocks # Update java doc for allocateBlock in ScmBlockLocationProtocolClientSideTransalatorPB.java and ScmBlockLocationProtocol.java # Setting numBlocks is missing when constructing request object in allocateBlock in ScmBlockLocationProtocolClientSideTransalatorPB.java. # And one more question, do we need to pass this block size from OM to SCM, as we read from config "ozone.scm.block.size” in OM. > OzoneManager#openKey should do multiple block allocations in a single SCM rpc > call > -- > > Key: HDDS-1095 > URL: https://issues.apache.org/jira/browse/HDDS-1095 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1095.001.patch > > > Currently in KeyManagerImpl#openKey, for a large key allocation, multiple > blocks are allocated in different rpc calls. If the key length is already > known, then multiple blocks can be allocated in one rpc call to SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1101) SCM CA: Write Certificate information to SCM Metadata
[ https://issues.apache.org/jira/browse/HDDS-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772221#comment-16772221 ] Ajay Kumar commented on HDDS-1101: -- [~xyao],[~anu] lets make that change in HDDS-1060. +1 > SCM CA: Write Certificate information to SCM Metadata > - > > Key: HDDS-1101 > URL: https://issues.apache.org/jira/browse/HDDS-1101 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Major > Attachments: HDDS-1101.000.patch, HDDS-1101.001.patch > > > Make SCM CA write to the Metadata layer of SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1135) Ozone jars are missing in the Ozone Snapshot tar
[
https://issues.apache.org/jira/browse/HDDS-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772236#comment-16772236
]
Dinesh Chitlangia commented on HDDS-1135:
-
[~shashikant] thanks for filing the jira.
[~elek] thanks for the inputs, that was helpful.
Attached patch 00 for review.
I verified manually by building tar and the extracting it.
> Ozone jars are missing in the Ozone Snapshot tar
>
>
> Key: HDDS-1135
> URL: https://issues.apache.org/jira/browse/HDDS-1135
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Dinesh Chitlangia
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1135.00.patch
>
>
> After executing an ozone dist build the library jars are missing from the
> created tar file.
> The problem is on the maven side. The tar file creation is called before the
> jar copies.
> {code:java}
> cd hadoop-ozone/dist
> mvn clean package | grep "\-\-\-"{code}
> {code:java}
> [INFO] < org.apache.hadoop:hadoop-ozone-dist
> >-
> [INFO] [ pom
> ]-
> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-ozone-dist
> ---
> [INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-ozone-dist
> ---
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @
> hadoop-ozone-dist ---
> [INFO] --- exec-maven-plugin:1.3.1:exec (dist) @ hadoop-ozone-dist ---
> [INFO] --- exec-maven-plugin:1.3.1:exec (tar-ozone) @ hadoop-ozone-dist ---
> [INFO] --- maven-site-plugin:3.6:attach-descriptor (attach-descriptor) @
> hadoop-ozone-dist ---
> [INFO] --- maven-dependency-plugin:3.0.2:build-classpath
> (add-classpath-descriptor) @ hadoop-ozone-dist ---
> [INFO] --- maven-dependency-plugin:3.0.2:copy (copy-classpath-files) @
> hadoop-ozone-dist ---
> [INFO] --- maven-dependency-plugin:3.0.2:copy-dependencies (copy-jars) @
> hadoop-ozone-dist ---
> [INFO] --- maven-jar-plugin:2.5:test-jar (default) @ hadoop-ozone-dist
> ---{code}
> The right order of the plugin executions are:
> * Call 'dist' (dist-layout-stitching, it cleans the destination directory)
> * Copy the jar files (copy-classpath-files, copy-jars)
> * Create the tar package (tar-ozone)
> It could be done with adjusting the maven phases in the pom.xml
> I would suggest to move 'dist' to the 'compile' phase, move
> 'copy-classpath-files' and 'copy-jars' to the 'prepare-package' phase, and
> keep 'tar-ozone' at the 'package' phase.
> With this setup we can be sure that the steps are executed in the right order.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1135) Ozone jars are missing in the Ozone Snapshot tar
[
https://issues.apache.org/jira/browse/HDDS-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Chitlangia updated HDDS-1135:
Attachment: HDDS-1135.00.patch
Status: Patch Available (was: Open)
> Ozone jars are missing in the Ozone Snapshot tar
>
>
> Key: HDDS-1135
> URL: https://issues.apache.org/jira/browse/HDDS-1135
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Dinesh Chitlangia
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1135.00.patch
>
>
> After executing an ozone dist build the library jars are missing from the
> created tar file.
> The problem is on the maven side. The tar file creation is called before the
> jar copies.
> {code:java}
> cd hadoop-ozone/dist
> mvn clean package | grep "\-\-\-"{code}
> {code:java}
> [INFO] < org.apache.hadoop:hadoop-ozone-dist
> >-
> [INFO] [ pom
> ]-
> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-ozone-dist
> ---
> [INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-ozone-dist
> ---
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @
> hadoop-ozone-dist ---
> [INFO] --- exec-maven-plugin:1.3.1:exec (dist) @ hadoop-ozone-dist ---
> [INFO] --- exec-maven-plugin:1.3.1:exec (tar-ozone) @ hadoop-ozone-dist ---
> [INFO] --- maven-site-plugin:3.6:attach-descriptor (attach-descriptor) @
> hadoop-ozone-dist ---
> [INFO] --- maven-dependency-plugin:3.0.2:build-classpath
> (add-classpath-descriptor) @ hadoop-ozone-dist ---
> [INFO] --- maven-dependency-plugin:3.0.2:copy (copy-classpath-files) @
> hadoop-ozone-dist ---
> [INFO] --- maven-dependency-plugin:3.0.2:copy-dependencies (copy-jars) @
> hadoop-ozone-dist ---
> [INFO] --- maven-jar-plugin:2.5:test-jar (default) @ hadoop-ozone-dist
> ---{code}
> The right order of the plugin executions are:
> * Call 'dist' (dist-layout-stitching, it cleans the destination directory)
> * Copy the jar files (copy-classpath-files, copy-jars)
> * Create the tar package (tar-ozone)
> It could be done with adjusting the maven phases in the pom.xml
> I would suggest to move 'dist' to the 'compile' phase, move
> 'copy-classpath-files' and 'copy-jars' to the 'prepare-package' phase, and
> keep 'tar-ozone' at the 'package' phase.
> With this setup we can be sure that the steps are executed in the right order.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14052) RBF: Use Router keytab for WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772234#comment-16772234
]
Brahma Reddy Battula commented on HDFS-14052:
-
For testing router startup with security should be fine.(having principle in
keytab and without having).
> RBF: Use Router keytab for WebHDFS
> --
>
> Key: HDFS-14052
> URL: https://issues.apache.org/jira/browse/HDFS-14052
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14052-HDFS-13891.0.patch
>
>
> When the RouterHttpServer starts it does:
> {code}
> NameNodeHttpServer.initWebHdfs(conf, httpAddress.getHostName(),
> httpServer,
> RouterWebHdfsMethods.class.getPackage().getName());
> {code}
> This function is in the NN and is pretty generic.
> However, it then calls to NameNodeHttpServer#getAuthFilterParams, which does:
> {code}
> String httpKeytab = conf.get(DFSUtil.getSpnegoKeytabKey(conf,
> DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY));
> {code}
> In most cases, the regular web keytab will kick in, but we should make this a
> parameter and load the Router one just in case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13972) RBF: Support for Delegation Token (WebHDFS)
[ https://issues.apache.org/jira/browse/HDFS-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772229#comment-16772229 ] Brahma Reddy Battula commented on HDFS-13972: - [~crh] can you update the patch..? > RBF: Support for Delegation Token (WebHDFS) > --- > > Key: HDFS-13972 > URL: https://issues.apache.org/jira/browse/HDFS-13972 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: HDFS-13972-HDFS-13891.001.patch > > > HDFS Router should support issuing HDFS delegation tokens through WebHDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1101) SCM CA: Write Certificate information to SCM Metadata
[ https://issues.apache.org/jira/browse/HDDS-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772227#comment-16772227 ] Hudson commented on HDDS-1101: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #15995 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15995/]) HDDS-1101. SCM CA: Write Certificate information to SCM Metadata. (xyao: rev cf1a66d0da32b86ea69d5817df1aba0294fa66f9) * (edit) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/codec/OmBucketInfoCodec.java * (edit) hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/authority/TestDefaultCAServer.java * (edit) hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/codec/TestOmMultipartKeyInfoCodec.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/LongCodec.java * (add) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/CertificateStore.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TableIterator.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/SCMMetadataStoreRDBImpl.java * (edit) hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestRDBTableStore.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/DeletedBlocksTransactionCodec.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/StringCodec.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/utils/CertificateCodec.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/CodecRegistry.java * (edit) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/codec/VolumeListCodec.java * (edit) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/codec/OmVolumeArgsCodec.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/DefaultCAServer.java * (edit) hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestTypedRDBTableStore.java * (add) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/BigIntegerCodec.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java * (edit) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/codec/OmKeyInfoCodec.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/Table.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/SCMMetadataStore.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/DefaultApprover.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/Codec.java * (add) hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/authority/MockCAStore.java * (add) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/X509CertificateCodec.java * (edit) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/codec/OmMultipartKeyInfoCodec.java * (add) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMCertStore.java > SCM CA: Write Certificate information to SCM Metadata > - > > Key: HDDS-1101 > URL: https://issues.apache.org/jira/browse/HDDS-1101 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1101.000.patch, HDDS-1101.001.patch > > > Make SCM CA write to the Metadata layer of SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1101) SCM CA: Write Certificate information to SCM Metadata
[ https://issues.apache.org/jira/browse/HDDS-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDDS-1101: - Resolution: Fixed Fix Version/s: 0.4.0 Status: Resolved (was: Patch Available) Thanks [~anu] for the contribution and all for the reviews. I've commit the patch to trunk. > SCM CA: Write Certificate information to SCM Metadata > - > > Key: HDDS-1101 > URL: https://issues.apache.org/jira/browse/HDDS-1101 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1101.000.patch, HDDS-1101.001.patch > > > Make SCM CA write to the Metadata layer of SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13955) RBF: Support secure Namenode in NamenodeHeartbeatService
[ https://issues.apache.org/jira/browse/HDFS-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772218#comment-16772218 ] Brahma Reddy Battula commented on HDFS-13955: - [~surendrasingh] thanks for taking look. [~zhengxg3] shall I assign this issue to me..? > RBF: Support secure Namenode in NamenodeHeartbeatService > > > Key: HDFS-13955 > URL: https://issues.apache.org/jira/browse/HDFS-13955 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > Attachments: HDFS-13955-HDFS-13532.000.patch, > HDFS-13955-HDFS-13532.001.patch > > > Currently, the NamenodeHeartbeatService uses JMX to get the metrics from the > Namenodes. We should support HTTPs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13123) RBF: Add a balancer tool to move data across subcluster
[ https://issues.apache.org/jira/browse/HDFS-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772199#comment-16772199 ] Brahma Reddy Battula commented on HDFS-13123: - [~ywskycn] any update on this..? shall we plan under HDFS-13891? > RBF: Add a balancer tool to move data across subcluster > > > Key: HDFS-13123 > URL: https://issues.apache.org/jira/browse/HDFS-13123 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Major > Attachments: HDFS Router-Based Federation Rebalancer.pdf > > > Follow the discussion in HDFS-12615. This Jira is to track effort for > building a rebalancer tool, used by router-based federation to move data > among subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-3246) pRead equivalent for direct read path
[ https://issues.apache.org/jira/browse/HDFS-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772192#comment-16772192 ] Wei-Chiu Chuang edited comment on HDFS-3246 at 2/19/19 6:18 PM: Hi [~zhangchen], I assigned this jira to Sahil. Please Feel free to contribute by reviewing the patch too. Your contribution is greatly appreciated. was (Author: jojochuang): Hi [~zhangchen], I assigned this jira to Sahil. Please Feel free to contribute by reviewing the patch too. > pRead equivalent for direct read path > - > > Key: HDFS-3246 > URL: https://issues.apache.org/jira/browse/HDFS-3246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, performance >Affects Versions: 3.0.0-alpha1 >Reporter: Henry Robinson >Assignee: Sahil Takiar >Priority: Major > Attachments: HDFS-3246.001.patch > > > There is no pread equivalent in ByteBufferReadable. We should consider adding > one. It would be relatively easy to implement for the distributed case > (certainly compared to HDFS-2834), since DFSInputStream does most of the > heavy lifting. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-3246) pRead equivalent for direct read path
[ https://issues.apache.org/jira/browse/HDFS-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772192#comment-16772192 ] Wei-Chiu Chuang commented on HDFS-3246: --- Hi [~zhangchen], I assigned this jira to Sahil. Please Feel free to contribute by reviewing the patch too. > pRead equivalent for direct read path > - > > Key: HDFS-3246 > URL: https://issues.apache.org/jira/browse/HDFS-3246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, performance >Affects Versions: 3.0.0-alpha1 >Reporter: Henry Robinson >Assignee: Sahil Takiar >Priority: Major > Attachments: HDFS-3246.001.patch > > > There is no pread equivalent in ByteBufferReadable. We should consider adding > one. It would be relatively easy to implement for the distributed case > (certainly compared to HDFS-2834), since DFSInputStream does most of the > heavy lifting. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
