[jira] [Updated] (HDFS-14824) [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the benchmark results.
[ https://issues.apache.org/jira/browse/HDFS-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-14824: Status: Patch Available (was: Open) Sent a PR. I almost just merged [this original PR|https://github.com/linkedin/dynamometer/pull/76], fixed the license and checkstyle warns. > [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the > benchmark results. > --- > > Key: HDFS-14824 > URL: https://issues.apache.org/jira/browse/HDFS-14824 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Soya Miyoshi >Assignee: Takanobu Asanuma >Priority: Major > > According to the latest > [document|https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-dynamometer/Dynamometer.html > ], the benchmark results will be written in `Dauditreplay.output-path`. > However, current org.apache.hadooop.tools hasn't merged [this pull > request|https://github.com/linkedin/dynamometer/pull/76 ], so it does not > output the benchmark results. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14891) RBF: namenode links in NameFederation Health page (federationhealth.html) cannot use https scheme
[
https://issues.apache.org/jira/browse/HDFS-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962714#comment-16962714
]
Hadoop QA commented on HDFS-14891:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
47s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 1s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterRpc |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14891 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12984334/HDFS-14891.006.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle cc |
| uname | Linux 2969b1bd1903 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 012756a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/28209/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28209/testReport/ |
| Max. process+thread count | 2761 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console
[jira] [Updated] (HDDS-2243) Fix docs for Running with HDFS section
[
https://issues.apache.org/jira/browse/HDDS-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Chitlangia updated HDDS-2243:
Labels: newbie (was: )
> Fix docs for Running with HDFS section
> --
>
> Key: HDDS-2243
> URL: https://issues.apache.org/jira/browse/HDDS-2243
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: documentation
>Affects Versions: 0.4.0
>Reporter: Dinesh Chitlangia
>Assignee: Dinesh Chitlangia
>Priority: Major
> Labels: newbie
>
> HDDS-4 says Fixed Version {{0.4.0}} but the 0.4.0 docs say Ozone won't work
> with secure clusters until HDDS-4 is fixed.
> https://hadoop.apache.org/ozone/docs/0.4.0-alpha/runningwithhdfs.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2243) Fix docs for Running with HDFS section
[
https://issues.apache.org/jira/browse/HDDS-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Chitlangia reassigned HDDS-2243:
---
Assignee: (was: Dinesh Chitlangia)
> Fix docs for Running with HDFS section
> --
>
> Key: HDDS-2243
> URL: https://issues.apache.org/jira/browse/HDDS-2243
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: documentation
>Affects Versions: 0.4.0
>Reporter: Dinesh Chitlangia
>Priority: Major
> Labels: newbie
>
> HDDS-4 says Fixed Version {{0.4.0}} but the 0.4.0 docs say Ozone won't work
> with secure clusters until HDDS-4 is fixed.
> https://hadoop.apache.org/ozone/docs/0.4.0-alpha/runningwithhdfs.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-426) Add field modificationTime for Volume and Bucket
[
https://issues.apache.org/jira/browse/HDDS-426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962671#comment-16962671
]
Anu Engineer commented on HDDS-426:
---
[~arp] Thanks for catching that, reopening since this JIRA is for modification
time. not creation time.
> Add field modificationTime for Volume and Bucket
>
>
> Key: HDDS-426
> URL: https://issues.apache.org/jira/browse/HDDS-426
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Manager
>Reporter: Dinesh Chitlangia
>Assignee: YiSheng Lien
>Priority: Major
> Labels: newbie
>
> There are update operations that can be performed for Volume, Bucket and Key.
> While Key records the modification time, Volume and & Bucket do not capture
> this.
>
> This Jira proposes to add the required field to Volume and Bucket in order to
> capture the modficationTime.
>
> Current Status:
> {noformat}
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoVolume /dummyvol
> 2018-09-10 17:16:12 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "owner" : {
> "name" : "bilbo"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "dummyvol",
> "createdOn" : "Mon, 10 Sep 2018 17:11:32 GMT",
> "createdBy" : "bilbo"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoBucket /dummyvol/mybuck
> 2018-09-10 17:15:25 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "volumeName" : "dummyvol",
> "bucketName" : "mybuck",
> "createdOn" : "Mon, 10 Sep 2018 17:12:09 GMT",
> "acls" : [ {
> "type" : "USER",
> "name" : "hadoop",
> "rights" : "READ_WRITE"
> }, {
> "type" : "GROUP",
> "name" : "users",
> "rights" : "READ_WRITE"
> }, {
> "type" : "USER",
> "name" : "spark",
> "rights" : "READ_WRITE"
> } ],
> "versioning" : "DISABLED",
> "storageType" : "DISK"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoKey /dummyvol/mybuck/myk1
> 2018-09-10 17:19:43 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "modifiedOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "size" : 0,
> "keyName" : "myk1",
> "keyLocations" : [ ]
> }{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2331) Client OOME due to buffer retention
[
https://issues.apache.org/jira/browse/HDDS-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962669#comment-16962669
]
Jitendra Nath Pandey commented on HDDS-2331:
Shall we resolve it, given RATIS-726 is committed, and HDDS-2375 tracks the
changes to optimize buffer allocation?
> Client OOME due to buffer retention
> ---
>
> Key: HDDS-2331
> URL: https://issues.apache.org/jira/browse/HDDS-2331
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.5.0
>Reporter: Attila Doroszlai
>Assignee: Shashikant Banerjee
>Priority: Critical
> Attachments: profiler.png
>
>
> Freon random key generator exhausts default heap after just few hundred 1MB
> keys. Heap dump on OOME reveals 150+ instances of
> {{ContainerCommandRequestMessage}}, each with 16MB {{byte[]}}.
> Steps to reproduce:
> # Start Ozone cluster with 1 datanode
> # Start Freon (5K keys of size 1MB)
> Result: OOME after a few hundred keys
> {noformat}
> $ cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone
> $ docker-compose up -d
> $ docker-compose exec scm bash
> $ export HADOOP_OPTS='-XX:+HeapDumpOnOutOfMemoryError'
> $ ozone freon rk --numOfThreads 1 --numOfVolumes 1 --numOfBuckets 1
> --replicationType RATIS --factor ONE --keySize 1048576 --numOfKeys 5120
> --bufferSize 65536
> ...
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid289.hprof ...
> Heap dump file created [1456141975 bytes in 7.760 secs]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDDS-426) Add field modificationTime for Volume and Bucket
[
https://issues.apache.org/jira/browse/HDDS-426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer reopened HDDS-426:
---
> Add field modificationTime for Volume and Bucket
>
>
> Key: HDDS-426
> URL: https://issues.apache.org/jira/browse/HDDS-426
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Manager
>Reporter: Dinesh Chitlangia
>Assignee: YiSheng Lien
>Priority: Major
> Labels: newbie
>
> There are update operations that can be performed for Volume, Bucket and Key.
> While Key records the modification time, Volume and & Bucket do not capture
> this.
>
> This Jira proposes to add the required field to Volume and Bucket in order to
> capture the modficationTime.
>
> Current Status:
> {noformat}
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoVolume /dummyvol
> 2018-09-10 17:16:12 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "owner" : {
> "name" : "bilbo"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "dummyvol",
> "createdOn" : "Mon, 10 Sep 2018 17:11:32 GMT",
> "createdBy" : "bilbo"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoBucket /dummyvol/mybuck
> 2018-09-10 17:15:25 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "volumeName" : "dummyvol",
> "bucketName" : "mybuck",
> "createdOn" : "Mon, 10 Sep 2018 17:12:09 GMT",
> "acls" : [ {
> "type" : "USER",
> "name" : "hadoop",
> "rights" : "READ_WRITE"
> }, {
> "type" : "GROUP",
> "name" : "users",
> "rights" : "READ_WRITE"
> }, {
> "type" : "USER",
> "name" : "spark",
> "rights" : "READ_WRITE"
> } ],
> "versioning" : "DISABLED",
> "storageType" : "DISK"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoKey /dummyvol/mybuck/myk1
> 2018-09-10 17:19:43 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "modifiedOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "size" : 0,
> "keyName" : "myk1",
> "keyLocations" : [ ]
> }{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1847) Datanode Kerberos principal and keytab config key looks inconsistent
[ https://issues.apache.org/jira/browse/HDDS-1847?focusedWorklogId=335953=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-335953 ] ASF GitHub Bot logged work on HDDS-1847: Author: ASF GitHub Bot Created on: 30/Oct/19 04:10 Start Date: 30/Oct/19 04:10 Worklog Time Spent: 10m Work Description: anuengineer commented on issue #1678: HDDS-1847: Datanode Kerberos principal and keytab config key looks inconsistent URL: https://github.com/apache/hadoop/pull/1678#issuecomment-547731434 +1. I will commit this when @christeoh gets a chance to rebase this to hadoop-ozone git repo. He is working on the older Hadoop Repo still. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 335953) Time Spent: 50m (was: 40m) > Datanode Kerberos principal and keytab config key looks inconsistent > > > Key: HDDS-1847 > URL: https://issues.apache.org/jira/browse/HDDS-1847 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.5.0 >Reporter: Eric Yang >Assignee: Chris Teoh >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Ozone Kerberos configuration can be very confusing: > | config name | Description | > | hdds.scm.kerberos.principal | SCM service principal | > | hdds.scm.kerberos.keytab.file | SCM service keytab file | > | ozone.om.kerberos.principal | Ozone Manager service principal | > | ozone.om.kerberos.keytab.file | Ozone Manager keytab file | > | hdds.scm.http.kerberos.principal | SCM service spnego principal | > | hdds.scm.http.kerberos.keytab.file | SCM service spnego keytab file | > | ozone.om.http.kerberos.principal | Ozone Manager spnego principal | > | ozone.om.http.kerberos.keytab.file | Ozone Manager spnego keytab file | > | hdds.datanode.http.kerberos.keytab | Datanode spnego keytab file | > | hdds.datanode.http.kerberos.principal | Datanode spnego principal | > | dfs.datanode.kerberos.principal | Datanode service principal | > | dfs.datanode.keytab.file | Datanode service keytab file | > The prefix are very different for each of the datanode configuration. It > would be nice to have some consistency for datanode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2379) OM terminates with RocksDB error while continuously writing keys.
[
https://issues.apache.org/jira/browse/HDDS-2379?focusedWorklogId=335951=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-335951
]
ASF GitHub Bot logged work on HDDS-2379:
Author: ASF GitHub Bot
Created on: 30/Oct/19 04:09
Start Date: 30/Oct/19 04:09
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #100: HDDS-2379.
OM terminates with RocksDB error while continuously writing keys.
URL: https://github.com/apache/hadoop-ozone/pull/100
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 335951)
Time Spent: 20m (was: 10m)
> OM terminates with RocksDB error while continuously writing keys.
> -
>
> Key: HDDS-2379
> URL: https://issues.apache.org/jira/browse/HDDS-2379
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Bharat Viswanadham
>Priority: Critical
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Exception trace after writing around 800,000 keys.
> {code}
> 2019-10-29 11:15:15,131 ERROR
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer: Terminating with
> exit status 1: During flush to DB encountered err
> or in OMDoubleBuffer flush thread OMDoubleBufferFlushThread
> java.io.IOException: Unable to write the batch.
> at
> org.apache.hadoop.hdds.utils.db.RDBBatchOperation.commit(RDBBatchOperation.java:48)
> at
> org.apache.hadoop.hdds.utils.db.RDBStore.commitBatchOperation(RDBStore.java:240)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:146)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.rocksdb.RocksDBException: unknown WriteBatch tag
> at org.rocksdb.RocksDB.write0(Native Method)
> at org.rocksdb.RocksDB.write(RocksDB.java:1421)
> at
> org.apache.hadoop.hdds.utils.db.RDBBatchOperation.commit(RDBBatchOperation.java:46)
> ... 3 more
> {code}
> Assigning to [~bharat] since he has already started work on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14891) RBF: namenode links in NameFederation Health page (federationhealth.html) cannot use https scheme
[ https://issues.apache.org/jira/browse/HDFS-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962666#comment-16962666 ] Xieming Li edited comment on HDFS-14891 at 10/30/19 4:06 AM: - HI, [~elgoiri], [~tasanuma]. I have updated patch: - Added test for webScheme - Added constant in TestStateStoreMembershipState - Suppressed CheckStyle warning was (Author: risyomei): [~elgoiri][~tasanuma] I have updated patch: - Added test for webScheme - Added constant in TestStateStoreMembershipState - Suppressed CheckStyle warning > RBF: namenode links in NameFederation Health page (federationhealth.html) > cannot use https scheme > -- > > Key: HDFS-14891 > URL: https://issues.apache.org/jira/browse/HDFS-14891 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf, ui >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Major > Attachments: HDFS-14891.001.patch, HDFS-14891.002.patch, > HDFS-14891.003.patch, HDFS-14891.004.patch, HDFS-14891.005.patch, > HDFS-14891.006.patch, HDFS-14891.patch > > > The scheme of links in federationhealth.html are hard coded as 'http'. > It should be set to 'https' when dfs.http.policy is HTTPS_ONLY > (HTTP_AND_HTTPS also, maybe) > > [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L168-L169] > [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L236] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14891) RBF: namenode links in NameFederation Health page (federationhealth.html) cannot use https scheme
[ https://issues.apache.org/jira/browse/HDFS-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962666#comment-16962666 ] Xieming Li commented on HDFS-14891: --- [~elgoiri][~tasanuma] I have updated patch: - Added test for webScheme - Added constant in TestStateStoreMembershipState - Suppressed CheckStyle warning > RBF: namenode links in NameFederation Health page (federationhealth.html) > cannot use https scheme > -- > > Key: HDFS-14891 > URL: https://issues.apache.org/jira/browse/HDFS-14891 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf, ui >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Major > Attachments: HDFS-14891.001.patch, HDFS-14891.002.patch, > HDFS-14891.003.patch, HDFS-14891.004.patch, HDFS-14891.005.patch, > HDFS-14891.006.patch, HDFS-14891.patch > > > The scheme of links in federationhealth.html are hard coded as 'http'. > It should be set to 'https' when dfs.http.policy is HTTPS_ONLY > (HTTP_AND_HTTPS also, maybe) > > [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L168-L169] > [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L236] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14891) RBF: namenode links in NameFederation Health page (federationhealth.html) cannot use https scheme
[ https://issues.apache.org/jira/browse/HDFS-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xieming Li updated HDFS-14891: -- Attachment: HDFS-14891.006.patch Status: Patch Available (was: Open) > RBF: namenode links in NameFederation Health page (federationhealth.html) > cannot use https scheme > -- > > Key: HDFS-14891 > URL: https://issues.apache.org/jira/browse/HDFS-14891 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf, ui >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Major > Attachments: HDFS-14891.001.patch, HDFS-14891.002.patch, > HDFS-14891.003.patch, HDFS-14891.004.patch, HDFS-14891.005.patch, > HDFS-14891.006.patch, HDFS-14891.patch > > > The scheme of links in federationhealth.html are hard coded as 'http'. > It should be set to 'https' when dfs.http.policy is HTTPS_ONLY > (HTTP_AND_HTTPS also, maybe) > > [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L168-L169] > [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L236] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14891) RBF: namenode links in NameFederation Health page (federationhealth.html) cannot use https scheme
[ https://issues.apache.org/jira/browse/HDFS-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xieming Li updated HDFS-14891: -- Status: Open (was: Patch Available) > RBF: namenode links in NameFederation Health page (federationhealth.html) > cannot use https scheme > -- > > Key: HDFS-14891 > URL: https://issues.apache.org/jira/browse/HDFS-14891 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf, ui >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Major > Attachments: HDFS-14891.001.patch, HDFS-14891.002.patch, > HDFS-14891.003.patch, HDFS-14891.004.patch, HDFS-14891.005.patch, > HDFS-14891.patch > > > The scheme of links in federationhealth.html are hard coded as 'http'. > It should be set to 'https' when dfs.http.policy is HTTPS_ONLY > (HTTP_AND_HTTPS also, maybe) > > [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L168-L169] > [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L236] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work stopped] (HDFS-14718) HttpFS: Sort response by key names as WebHDFS does
[ https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-14718 stopped by Siyao Meng. - > HttpFS: Sort response by key names as WebHDFS does > -- > > Key: HDFS-14718 > URL: https://issues.apache.org/jira/browse/HDFS-14718 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > *Example* > See description of HDFS-14665 for an example of LISTSTATUS. > *Analysis* > WebHDFS is [using a > TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] > to serialize HdfsFileStatus, while HttpFS [uses a > LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] > to serialize FileStatus. > *Questions* > Why the difference? Is this intentional? > - I looked into the Git history. It seems it's simply because WebHDFS uses > TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. > It is not only limited to LISTSTATUS, but ALL other request's JSON > serialization. > Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap > in HttpFS serialization in FSOperations class? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14920) Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission
[
https://issues.apache.org/jira/browse/HDFS-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962657#comment-16962657
]
Hadoop QA commented on HDFS-14920:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 26s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}145m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14920 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12984317/HDFS-14920.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7a4e981d6a66 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 012756a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/28208/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28208/testReport/ |
| Max. process+thread count | 4003 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28208/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was
[jira] [Commented] (HDFS-14927) RBF: Add metrics for async callers thread pool
[ https://issues.apache.org/jira/browse/HDFS-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962646#comment-16962646 ] Íñigo Goiri commented on HDFS-14927: [^HDFS-14927.007.patch]. [~ayushtkn] do you mind taking a second look? > RBF: Add metrics for async callers thread pool > -- > > Key: HDFS-14927 > URL: https://issues.apache.org/jira/browse/HDFS-14927 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Minor > Attachments: HDFS-14927.001.patch, HDFS-14927.002.patch, > HDFS-14927.003.patch, HDFS-14927.004.patch, HDFS-14927.005.patch, > HDFS-14927.006.patch, HDFS-14927.007.patch > > > It is good to add some monitoring on the async caller thread pool to handle > fan-out RPC client requests, so we know the utilization and when to bump up > dfs.federation.router.client.thread-size -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14927) RBF: Add metrics for async callers thread pool
[
https://issues.apache.org/jira/browse/HDFS-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962635#comment-16962635
]
Hadoop QA commented on HDFS-14927:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 49s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
57s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14927 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12984316/HDFS-14927.007.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d6c2a5ef295b 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 012756a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28207/testReport/ |
| Max. process+thread count | 2729 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28207/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Add metrics for async callers thread pool
> --
>
> Key: HDFS-14927
> URL:
[jira] [Commented] (HDFS-14824) [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the benchmark results.
[ https://issues.apache.org/jira/browse/HDFS-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962616#comment-16962616 ] Takanobu Asanuma commented on HDFS-14824: - Thanks for the confirmation. I'd like to assign this jira to myself. > [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the > benchmark results. > --- > > Key: HDFS-14824 > URL: https://issues.apache.org/jira/browse/HDFS-14824 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Soya Miyoshi >Priority: Major > > According to the latest > [document|https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-dynamometer/Dynamometer.html > ], the benchmark results will be written in `Dauditreplay.output-path`. > However, current org.apache.hadooop.tools hasn't merged [this pull > request|https://github.com/linkedin/dynamometer/pull/76 ], so it does not > output the benchmark results. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14824) [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the benchmark results.
[ https://issues.apache.org/jira/browse/HDFS-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma reassigned HDFS-14824: --- Assignee: Takanobu Asanuma > [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the > benchmark results. > --- > > Key: HDFS-14824 > URL: https://issues.apache.org/jira/browse/HDFS-14824 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Soya Miyoshi >Assignee: Takanobu Asanuma >Priority: Major > > According to the latest > [document|https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-dynamometer/Dynamometer.html > ], the benchmark results will be written in `Dauditreplay.output-path`. > However, current org.apache.hadooop.tools hasn't merged [this pull > request|https://github.com/linkedin/dynamometer/pull/76 ], so it does not > output the benchmark results. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14907) [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
[
https://issues.apache.org/jira/browse/HDFS-14907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-14907:
Resolution: Fixed
Status: Resolved (was: Patch Available)
[~xkrogen] Thanks for reviewing and committing it!
> [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
> --
>
> Key: HDFS-14907
> URL: https://issues.apache.org/jira/browse/HDFS-14907
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.3.0
>
>
> When executing {{start-dynamometer-cluster.sh}} with Hadoop-3 binary,
> datanodes fail to run with the following log and
> {{start-dynamometer-cluster.sh}} fails.
> {noformat}
> LogType:stderr
> LogLastModifiedTime:Wed Oct 09 15:03:09 +0900 2019
> LogLength:1386
> LogContents:
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:299)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:243)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:252)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2982)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2972)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.formatDataNodeDirs(MiniDFSCluster.java:2834)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(SimulatedDataNodes.java:123)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.main(SimulatedDataNodes.java:88)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 9 more
> ./start-component.sh: line 317: kill: (2261) - No such process
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14927) RBF: Add metrics for async callers thread pool
[
https://issues.apache.org/jira/browse/HDFS-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962610#comment-16962610
]
Hadoop QA commented on HDFS-14927:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 44s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
37s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14927 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12984314/HDFS-14927.006.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5677db82e1a3 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 012756a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28206/testReport/ |
| Max. process+thread count | 2753 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28206/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Add metrics for async callers thread pool
> --
>
> Key: HDFS-14927
> URL:
[jira] [Comment Edited] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962583#comment-16962583 ] Bharat Viswanadham edited comment on HDDS-2356 at 10/30/19 1:23 AM: # When uploaded 2 parts, and when complete upload 1 part no error # During complete multipart upload name/part number not matching with uploaded part and part number then InvalidPart error # When parts are not specified in sorted order InvalidPartOrder # During complete multipart upload when no uploaded parts, and we specify some parts then also InvalidPart # Uploaded parts 1,2,3 and during complete we can do upload 1,3 (No error) # When part 3 uploaded, complete with part 3 can be done Found few scenarios where we behave differently need to handled in Ozone S3. And also if you can enable audit logs and see what is the parts we are getting during complete MPU and for that key what are parts available, then also we can see where the error occurs. But we should handle the above scenarios in Ozone. But the question is why goofys when uploading is uploading like this or we missed any other cases still. To answer this audit logs will help here. was (Author: bharatviswa): # When uploaded 2 parts, and when complete upload 1 part no error # During complete multipart upload name/part number not matching with uploaded part and part number then InvalidPart error # When parts are not specified in sorted order InvalidPartOrder # During complete multipart upload when no uploaded parts, and we specify some parts then also InvalidPart # Uploaded parts 1,2,3 and during complete we can do upload 1,3 (No error) # When part 3 uploaded, complete with part 3 can be done Found few scenarios where we behave differently need to handled in Ozone S3. And also if you can enable audit logs and see what is the parts we are getting during complete MPU and for that key what are parts available, then also we can see where the error occurs. But we should handle the above scenarios in Ozone. But the question is why goofys when uploading is uploading like this or we missed any other cases still. > Multipart upload report errors while writing to ozone Ratis pipeline > > > Key: HDDS-2356 > URL: https://issues.apache.org/jira/browse/HDDS-2356 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.4.1 > Environment: Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM > on a separate VM >Reporter: Li Cheng >Assignee: Bharat Viswanadham >Priority: Blocker > Fix For: 0.5.0 > > > Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say > it's VM0. > I use goofys as a fuse and enable ozone S3 gateway to mount ozone to a path > on VM0, while reading data from VM0 local disk and write to mount path. The > dataset has various sizes of files from 0 byte to GB-level and it has a > number of ~50,000 files. > The writing is slow (1GB for ~10 mins) and it stops after around 4GB. As I > look at hadoop-root-om-VM_50_210_centos.out log, I see OM throwing errors > related with Multipart upload. This error eventually causes the writing to > terminate and OM to be closed. > > 2019-10-24 16:01:59,527 [OMDoubleBufferFlushThread] ERROR - Terminating with > exit status 2: OMDoubleBuffer flush > threadOMDoubleBufferFlushThreadencountered Throwable error > java.util.ConcurrentModificationException > at java.util.TreeMap.forEach(TreeMap.java:1004) > at > org.apache.hadoop.ozone.om.helpers.OmMultipartKeyInfo.getProto(OmMultipartKeyInfo.java:111) > at > org.apache.hadoop.ozone.om.codec.OmMultipartKeyInfoCodec.toPersistedFormat(OmMultipartKeyInfoCodec.java:38) > at > org.apache.hadoop.ozone.om.codec.OmMultipartKeyInfoCodec.toPersistedFormat(OmMultipartKeyInfoCodec.java:31) > at > org.apache.hadoop.hdds.utils.db.CodecRegistry.asRawData(CodecRegistry.java:68) > at > org.apache.hadoop.hdds.utils.db.TypedTable.putWithBatch(TypedTable.java:125) > at > org.apache.hadoop.ozone.om.response.s3.multipart.S3MultipartUploadCommitPartResponse.addToDBBatch(S3MultipartUploadCommitPartResponse.java:112) > at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$flushTransactions$0(OzoneManagerDoubleBuffer.java:137) > at java.util.Iterator.forEachRemaining(Iterator.java:116) > at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:135) > at java.lang.Thread.run(Thread.java:745) > 2019-10-24 16:01:59,629 [shutdown-hook-0] INFO - SHUTDOWN_MSG: -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail:
[jira] [Comment Edited] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962583#comment-16962583 ] Bharat Viswanadham edited comment on HDDS-2356 at 10/30/19 1:23 AM: # When uploaded 2 parts, and when complete upload 1 part no error # During complete multipart upload name/part number not matching with uploaded part and part number then InvalidPart error # When parts are not specified in sorted order InvalidPartOrder # During complete multipart upload when no uploaded parts, and we specify some parts then also InvalidPart # Uploaded parts 1,2,3 and during complete we can do upload 1,3 (No error) # When part 3 uploaded, complete with part 3 can be done Found few scenarios where we behave differently need to handled in Ozone S3. And also if you can enable audit logs and see what is the parts we are getting during complete MPU and for that key what are parts available, then also we can see where the error occurs. But we should handle the above scenarios in Ozone. But the question is why goofys when uploading is uploading like this or we missed any other cases still. was (Author: bharatviswa): # When uploaded 2 parts, and when complete upload 1 part no error # During complete multipart upload name/part number not matching with uploaded part and part number then InvalidPart error # When parts are not specified in sorted order InvalidPartOrder # During complete multipart upload when no uploaded parts, and we specify some parts then also InvalidPart # Uploaded parts 1,2,3 and during complete we can do upload 1,3 (No error) # When part 3 uploaded, complete with part 3 can be done Found few scenarios where we behave differently need to handled in Ozone S3. And also if you can enable audit logs and see what is the parts we are getting during complete MPU and for that key what are parts available, then also we can see where the error occurs. But we should handle above scenarios in Ozone. > Multipart upload report errors while writing to ozone Ratis pipeline > > > Key: HDDS-2356 > URL: https://issues.apache.org/jira/browse/HDDS-2356 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.4.1 > Environment: Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM > on a separate VM >Reporter: Li Cheng >Assignee: Bharat Viswanadham >Priority: Blocker > Fix For: 0.5.0 > > > Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say > it's VM0. > I use goofys as a fuse and enable ozone S3 gateway to mount ozone to a path > on VM0, while reading data from VM0 local disk and write to mount path. The > dataset has various sizes of files from 0 byte to GB-level and it has a > number of ~50,000 files. > The writing is slow (1GB for ~10 mins) and it stops after around 4GB. As I > look at hadoop-root-om-VM_50_210_centos.out log, I see OM throwing errors > related with Multipart upload. This error eventually causes the writing to > terminate and OM to be closed. > > 2019-10-24 16:01:59,527 [OMDoubleBufferFlushThread] ERROR - Terminating with > exit status 2: OMDoubleBuffer flush > threadOMDoubleBufferFlushThreadencountered Throwable error > java.util.ConcurrentModificationException > at java.util.TreeMap.forEach(TreeMap.java:1004) > at > org.apache.hadoop.ozone.om.helpers.OmMultipartKeyInfo.getProto(OmMultipartKeyInfo.java:111) > at > org.apache.hadoop.ozone.om.codec.OmMultipartKeyInfoCodec.toPersistedFormat(OmMultipartKeyInfoCodec.java:38) > at > org.apache.hadoop.ozone.om.codec.OmMultipartKeyInfoCodec.toPersistedFormat(OmMultipartKeyInfoCodec.java:31) > at > org.apache.hadoop.hdds.utils.db.CodecRegistry.asRawData(CodecRegistry.java:68) > at > org.apache.hadoop.hdds.utils.db.TypedTable.putWithBatch(TypedTable.java:125) > at > org.apache.hadoop.ozone.om.response.s3.multipart.S3MultipartUploadCommitPartResponse.addToDBBatch(S3MultipartUploadCommitPartResponse.java:112) > at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$flushTransactions$0(OzoneManagerDoubleBuffer.java:137) > at java.util.Iterator.forEachRemaining(Iterator.java:116) > at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:135) > at java.lang.Thread.run(Thread.java:745) > 2019-10-24 16:01:59,629 [shutdown-hook-0] INFO - SHUTDOWN_MSG: -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962583#comment-16962583 ] Bharat Viswanadham commented on HDDS-2356: -- # When uploaded 2 parts, and when complete upload 1 part no error # During complete multipart upload name/part number not matching with uploaded part and part number then InvalidPart error # When parts are not specified in sorted order InvalidPartOrder # During complete multipart upload when no uploaded parts, and we specify some parts then also InvalidPart # Uploaded parts 1,2,3 and during complete we can do upload 1,3 (No error) # When part 3 uploaded, complete with part 3 can be done Found few scenarios where we behave differently need to handled in Ozone S3. And also if you can enable audit logs and see what is the parts we are getting during complete MPU and for that key what are parts available, then also we can see where the error occurs. But we should handle above scenarios in Ozone. > Multipart upload report errors while writing to ozone Ratis pipeline > > > Key: HDDS-2356 > URL: https://issues.apache.org/jira/browse/HDDS-2356 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.4.1 > Environment: Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM > on a separate VM >Reporter: Li Cheng >Assignee: Bharat Viswanadham >Priority: Blocker > Fix For: 0.5.0 > > > Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say > it's VM0. > I use goofys as a fuse and enable ozone S3 gateway to mount ozone to a path > on VM0, while reading data from VM0 local disk and write to mount path. The > dataset has various sizes of files from 0 byte to GB-level and it has a > number of ~50,000 files. > The writing is slow (1GB for ~10 mins) and it stops after around 4GB. As I > look at hadoop-root-om-VM_50_210_centos.out log, I see OM throwing errors > related with Multipart upload. This error eventually causes the writing to > terminate and OM to be closed. > > 2019-10-24 16:01:59,527 [OMDoubleBufferFlushThread] ERROR - Terminating with > exit status 2: OMDoubleBuffer flush > threadOMDoubleBufferFlushThreadencountered Throwable error > java.util.ConcurrentModificationException > at java.util.TreeMap.forEach(TreeMap.java:1004) > at > org.apache.hadoop.ozone.om.helpers.OmMultipartKeyInfo.getProto(OmMultipartKeyInfo.java:111) > at > org.apache.hadoop.ozone.om.codec.OmMultipartKeyInfoCodec.toPersistedFormat(OmMultipartKeyInfoCodec.java:38) > at > org.apache.hadoop.ozone.om.codec.OmMultipartKeyInfoCodec.toPersistedFormat(OmMultipartKeyInfoCodec.java:31) > at > org.apache.hadoop.hdds.utils.db.CodecRegistry.asRawData(CodecRegistry.java:68) > at > org.apache.hadoop.hdds.utils.db.TypedTable.putWithBatch(TypedTable.java:125) > at > org.apache.hadoop.ozone.om.response.s3.multipart.S3MultipartUploadCommitPartResponse.addToDBBatch(S3MultipartUploadCommitPartResponse.java:112) > at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$flushTransactions$0(OzoneManagerDoubleBuffer.java:137) > at java.util.Iterator.forEachRemaining(Iterator.java:116) > at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:135) > at java.lang.Thread.run(Thread.java:745) > 2019-10-24 16:01:59,629 [shutdown-hook-0] INFO - SHUTDOWN_MSG: -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14920) Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission
[
https://issues.apache.org/jira/browse/HDFS-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962570#comment-16962570
]
Fei Hui commented on HDFS-14920:
Upload v005 patch. Remove unused imports to fix checkstyle
Failed tests are unrelated.
> Erasure Coding: Decommission may hang If one or more datanodes are out of
> service during decommission
> ---
>
> Key: HDFS-14920
> URL: https://issues.apache.org/jira/browse/HDFS-14920
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-14920.001.patch, HDFS-14920.002.patch,
> HDFS-14920.003.patch, HDFS-14920.004.patch, HDFS-14920.005.patch
>
>
> Decommission test hangs in our clusters.
> Have seen the messages as follow
> {quote}
> 2019-10-22 15:58:51,514 TRACE
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager: Block
> blk_-9223372035600425840_372987973 numExpected=9, numLive=5
> 2019-10-22 15:58:51,514 INFO BlockStateChange: Block:
> blk_-9223372035600425840_372987973, Expected Replicas: 9, live replicas: 5,
> corrupt replicas: 0, decommissioned replicas: 0, decommissioning replicas: 4,
> maintenance replicas: 0, live entering maintenance replicas: 0, excess
> replicas: 0, Is Open File: false, Datanodes having this block:
> 10.255.43.57:50010 10.255.53.12:50010 10.255.63.12:50010 10.255.62.39:50010
> 10.255.37.36:50010 10.255.33.15:50010 10.255.69.29:50010 10.255.51.13:50010
> 10.255.64.15:50010 , Current Datanode: 10.255.69.29:50010, Is current
> datanode decommissioning: true, Is current datanode entering maintenance:
> false
> 2019-10-22 15:58:51,514 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager: Node
> 10.255.69.29:50010 still has 1 blocks to replicate before it is a candidate
> to finish Decommission In Progress
> {quote}
> After digging the source code and cluster log, guess it happens as follow
> steps.
> # Storage strategy is RS-6-3-1024k.
> # EC block b consists of b0, b1, b2, b3, b4, b5, b6, b7, b8, b0 is from
> datanode dn0, b1 is from datanode dn1, ...etc
> # At the beginning dn0 is in decommission progress, b0 is replicated
> successfully, and dn0 is staill in decommission progress.
> # Later b1, b2, b3 in decommission progress, and dn4 containing b4 is out of
> service, so need to reconstruct, and create ErasureCodingWork to do it, in
> the ErasureCodingWork, additionalReplRequired is 4
> # Because hasAllInternalBlocks is false, Will call
> ErasureCodingWork#addTaskToDatanode ->
> DatanodeDescriptor#addBlockToBeErasureCoded, and send
> BlockECReconstructionInfo task to Datanode
> # DataNode can not reconstruction the block because targets is 4, greater
> than 3( parity number).
> There is a problem as follow, from BlockManager.java#scheduleReconstruction
> {code}
> // should reconstruct all the internal blocks before scheduling
> // replication task for decommissioning node(s).
> if (additionalReplRequired - numReplicas.decommissioning() -
> numReplicas.liveEnteringMaintenanceReplicas() > 0) {
> additionalReplRequired = additionalReplRequired -
> numReplicas.decommissioning() -
> numReplicas.liveEnteringMaintenanceReplicas();
> }
> {code}
> Should reconstruction firstly and then replicate for decommissioning. Because
> numReplicas.decommissioning() is 4, and additionalReplRequired is 4, that's
> wrong,
> numReplicas.decommissioning() should be 3, it should exclude live replica.
> If so, additionalReplRequired will be 1, reconstruction will schedule as
> expected. After that, decommission goes on.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14920) Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission
[
https://issues.apache.org/jira/browse/HDFS-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fei Hui updated HDFS-14920:
---
Attachment: HDFS-14920.005.patch
> Erasure Coding: Decommission may hang If one or more datanodes are out of
> service during decommission
> ---
>
> Key: HDFS-14920
> URL: https://issues.apache.org/jira/browse/HDFS-14920
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-14920.001.patch, HDFS-14920.002.patch,
> HDFS-14920.003.patch, HDFS-14920.004.patch, HDFS-14920.005.patch
>
>
> Decommission test hangs in our clusters.
> Have seen the messages as follow
> {quote}
> 2019-10-22 15:58:51,514 TRACE
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager: Block
> blk_-9223372035600425840_372987973 numExpected=9, numLive=5
> 2019-10-22 15:58:51,514 INFO BlockStateChange: Block:
> blk_-9223372035600425840_372987973, Expected Replicas: 9, live replicas: 5,
> corrupt replicas: 0, decommissioned replicas: 0, decommissioning replicas: 4,
> maintenance replicas: 0, live entering maintenance replicas: 0, excess
> replicas: 0, Is Open File: false, Datanodes having this block:
> 10.255.43.57:50010 10.255.53.12:50010 10.255.63.12:50010 10.255.62.39:50010
> 10.255.37.36:50010 10.255.33.15:50010 10.255.69.29:50010 10.255.51.13:50010
> 10.255.64.15:50010 , Current Datanode: 10.255.69.29:50010, Is current
> datanode decommissioning: true, Is current datanode entering maintenance:
> false
> 2019-10-22 15:58:51,514 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager: Node
> 10.255.69.29:50010 still has 1 blocks to replicate before it is a candidate
> to finish Decommission In Progress
> {quote}
> After digging the source code and cluster log, guess it happens as follow
> steps.
> # Storage strategy is RS-6-3-1024k.
> # EC block b consists of b0, b1, b2, b3, b4, b5, b6, b7, b8, b0 is from
> datanode dn0, b1 is from datanode dn1, ...etc
> # At the beginning dn0 is in decommission progress, b0 is replicated
> successfully, and dn0 is staill in decommission progress.
> # Later b1, b2, b3 in decommission progress, and dn4 containing b4 is out of
> service, so need to reconstruct, and create ErasureCodingWork to do it, in
> the ErasureCodingWork, additionalReplRequired is 4
> # Because hasAllInternalBlocks is false, Will call
> ErasureCodingWork#addTaskToDatanode ->
> DatanodeDescriptor#addBlockToBeErasureCoded, and send
> BlockECReconstructionInfo task to Datanode
> # DataNode can not reconstruction the block because targets is 4, greater
> than 3( parity number).
> There is a problem as follow, from BlockManager.java#scheduleReconstruction
> {code}
> // should reconstruct all the internal blocks before scheduling
> // replication task for decommissioning node(s).
> if (additionalReplRequired - numReplicas.decommissioning() -
> numReplicas.liveEnteringMaintenanceReplicas() > 0) {
> additionalReplRequired = additionalReplRequired -
> numReplicas.decommissioning() -
> numReplicas.liveEnteringMaintenanceReplicas();
> }
> {code}
> Should reconstruction firstly and then replicate for decommissioning. Because
> numReplicas.decommissioning() is 4, and additionalReplRequired is 4, that's
> wrong,
> numReplicas.decommissioning() should be 3, it should exclude live replica.
> If so, additionalReplRequired will be 1, reconstruction will schedule as
> expected. After that, decommission goes on.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14927) RBF: Add metrics for async callers thread pool
[ https://issues.apache.org/jira/browse/HDFS-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leon Gao updated HDFS-14927: Attachment: HDFS-14927.007.patch > RBF: Add metrics for async callers thread pool > -- > > Key: HDFS-14927 > URL: https://issues.apache.org/jira/browse/HDFS-14927 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Minor > Attachments: HDFS-14927.001.patch, HDFS-14927.002.patch, > HDFS-14927.003.patch, HDFS-14927.004.patch, HDFS-14927.005.patch, > HDFS-14927.006.patch, HDFS-14927.007.patch > > > It is good to add some monitoring on the async caller thread pool to handle > fan-out RPC client requests, so we know the utilization and when to bump up > dfs.federation.router.client.thread-size -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14927) RBF: Add metrics for async callers thread pool
[ https://issues.apache.org/jira/browse/HDFS-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962568#comment-16962568 ] Leon Gao commented on HDFS-14927: - Got it [~inigoiri], I was hesitating for that as well ^ Updated latest. > RBF: Add metrics for async callers thread pool > -- > > Key: HDFS-14927 > URL: https://issues.apache.org/jira/browse/HDFS-14927 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Minor > Attachments: HDFS-14927.001.patch, HDFS-14927.002.patch, > HDFS-14927.003.patch, HDFS-14927.004.patch, HDFS-14927.005.patch, > HDFS-14927.006.patch > > > It is good to add some monitoring on the async caller thread pool to handle > fan-out RPC client requests, so we know the utilization and when to bump up > dfs.federation.router.client.thread-size -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14927) RBF: Add metrics for async callers thread pool
[
https://issues.apache.org/jira/browse/HDFS-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962558#comment-16962558
]
Íñigo Goiri commented on HDFS-14927:
Thanks for [~LeonG].
A comment, in the waitFor, we shouldn't do asserts, it should be something like:
{code}
GenericTestUtils.waitFor(() -> {
Map newResult = objectMapper.readValue(
metrics.getAsyncCallerPool(), Map.class);
if (newResult.get("active") != 1) {
return false;
}
if (newResult.get("max") != 4) {
return false;
}
int total = newResult.get("total");
if (total >= 1 && total <= 4) {
return true;
}
return false;
}, 100, 2000);
{code}
> RBF: Add metrics for async callers thread pool
> --
>
> Key: HDFS-14927
> URL: https://issues.apache.org/jira/browse/HDFS-14927
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Leon Gao
>Assignee: Leon Gao
>Priority: Minor
> Attachments: HDFS-14927.001.patch, HDFS-14927.002.patch,
> HDFS-14927.003.patch, HDFS-14927.004.patch, HDFS-14927.005.patch,
> HDFS-14927.006.patch
>
>
> It is good to add some monitoring on the async caller thread pool to handle
> fan-out RPC client requests, so we know the utilization and when to bump up
> dfs.federation.router.client.thread-size
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14927) RBF: Add metrics for async callers thread pool
[ https://issues.apache.org/jira/browse/HDFS-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leon Gao updated HDFS-14927: Attachment: HDFS-14927.006.patch > RBF: Add metrics for async callers thread pool > -- > > Key: HDFS-14927 > URL: https://issues.apache.org/jira/browse/HDFS-14927 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Minor > Attachments: HDFS-14927.001.patch, HDFS-14927.002.patch, > HDFS-14927.003.patch, HDFS-14927.004.patch, HDFS-14927.005.patch, > HDFS-14927.006.patch > > > It is good to add some monitoring on the async caller thread pool to handle > fan-out RPC client requests, so we know the utilization and when to bump up > dfs.federation.router.client.thread-size -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2373) Move isUseRatis getFactor and getType from XCeiverClientManager
[ https://issues.apache.org/jira/browse/HDDS-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia resolved HDDS-2373. - Fix Version/s: 0.5.0 Resolution: Fixed > Move isUseRatis getFactor and getType from XCeiverClientManager > --- > > Key: HDDS-2373 > URL: https://issues.apache.org/jira/browse/HDDS-2373 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Istvan Fajth >Assignee: Istvan Fajth >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The given methods in XCeiverClientManager are working based on the > configuration supplied in the constructor of XCeiverClientManager class. > The only real code usage of this is in ContainerOperationsClient. > Refactor the ContainerOperationsClient constructor to work based on the > configuration, and then move these values there directly and set the values > of them at the constructor. Clean up all test references to the methods, and > remove the methods from the XCeiverClientManager. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2373) Move isUseRatis getFactor and getType from XCeiverClientManager
[ https://issues.apache.org/jira/browse/HDDS-2373?focusedWorklogId=335831=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-335831 ] ASF GitHub Bot logged work on HDDS-2373: Author: ASF GitHub Bot Created on: 29/Oct/19 23:56 Start Date: 29/Oct/19 23:56 Worklog Time Spent: 10m Work Description: dineshchitlangia commented on pull request #95: HDDS-2373 Move isUseRatis getFactor and getType from XCeiverClientManager URL: https://github.com/apache/hadoop-ozone/pull/95 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 335831) Time Spent: 20m (was: 10m) > Move isUseRatis getFactor and getType from XCeiverClientManager > --- > > Key: HDDS-2373 > URL: https://issues.apache.org/jira/browse/HDDS-2373 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Istvan Fajth >Assignee: Istvan Fajth >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The given methods in XCeiverClientManager are working based on the > configuration supplied in the constructor of XCeiverClientManager class. > The only real code usage of this is in ContainerOperationsClient. > Refactor the ContainerOperationsClient constructor to work based on the > configuration, and then move these values there directly and set the values > of them at the constructor. Clean up all test references to the methods, and > remove the methods from the XCeiverClientManager. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14941) Potential editlog race condition can cause corrupted file
[
https://issues.apache.org/jira/browse/HDFS-14941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962543#comment-16962543
]
Chen Liang commented on HDFS-14941:
---
ping [~shv], [~xkrogen], [~csun] FYI.
> Potential editlog race condition can cause corrupted file
> -
>
> Key: HDFS-14941
> URL: https://issues.apache.org/jira/browse/HDFS-14941
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
>
> Recently we encountered an issue that, after a failover, NameNode complains
> corrupted file/missing blocks. The blocks did recover after full block
> reports, so the blocks are not actually missing. After further investigation,
> we believe this is what happened:
> First of all, on SbN, it is possible that it receives block reports before
> corresponding edit tailing happened. In which case SbN postpones processing
> the DN block report, handled by the guarding logic below:
> {code:java}
> if (shouldPostponeBlocksFromFuture &&
> namesystem.isGenStampInFuture(iblk)) {
> queueReportedBlock(storageInfo, iblk, reportedState,
> QUEUE_REASON_FUTURE_GENSTAMP);
> continue;
> }
> {code}
> Basically if reported block has a future generation stamp, the DN report gets
> requeued.
> However, in {{FSNamesystem#storeAllocatedBlock}}, we have the following code:
> {code:java}
> // allocate new block, record block locations in INode.
> newBlock = createNewBlock();
> INodesInPath inodesInPath = INodesInPath.fromINode(pendingFile);
> saveAllocatedBlock(src, inodesInPath, newBlock, targets);
> persistNewBlock(src, pendingFile);
> offset = pendingFile.computeFileSize();
> {code}
> The line
> {{newBlock = createNewBlock();}}
> Would log an edit entry {{OP_SET_GENSTAMP_V2}} to bump generation stamp on
> Standby
> while the following line
> {{persistNewBlock(src, pendingFile);}}
> would log another edit entry {{OP_ADD_BLOCK}} to actually add the block on
> Standby.
> Then the race condition is that, imagine Standby has just processed
> {{OP_SET_GENSTAMP_V2}}, but not yet {{OP_ADD_BLOCK}} (if they just happen to
> be in different setment). Now a block report with new generation stamp comes
> in.
> Since the genstamp bump has already been processed, the reported block may
> not be considered as future block. So the guarding logic passes. But
> actually, the block hasn't been added to blockmap, because the second edit is
> yet to be tailed. So, the block then gets added to invalidate block list and
> we saw messages like:
> {code:java}
> BLOCK* addBlock: block XXX on node XXX size XXX does not belong to any file
> {code}
> Even worse, since this IBR is effectively lost, the NameNode has no
> information about this block, until the next full block report. So after a
> failover, the NN marks it as corrupt.
> This issue won't happen though, if both of the edit entries get tailed all
> together, so no IBR processing can happen in between. But in our case, we set
> edit tailing interval to super low (to allow Standby read), so when under
> high workload, there is a much much higher chance that the two entries are
> tailed separately, causing the issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14941) Potential editlog race condition can cause corrupted file
[
https://issues.apache.org/jira/browse/HDFS-14941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang reassigned HDFS-14941:
-
Assignee: Chen Liang
> Potential editlog race condition can cause corrupted file
> -
>
> Key: HDFS-14941
> URL: https://issues.apache.org/jira/browse/HDFS-14941
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
>
> Recently we encountered an issue that, after a failover, NameNode complains
> corrupted file/missing blocks. The blocks did recover after full block
> reports, so the blocks are not actually missing. After further investigation,
> we believe this is what happened:
> First of all, on SbN, it is possible that it receives block reports before
> corresponding edit tailing happened. In which case SbN postpones processing
> the DN block report, handled by the guarding logic below:
> {code:java}
> if (shouldPostponeBlocksFromFuture &&
> namesystem.isGenStampInFuture(iblk)) {
> queueReportedBlock(storageInfo, iblk, reportedState,
> QUEUE_REASON_FUTURE_GENSTAMP);
> continue;
> }
> {code}
> Basically if reported block has a future generation stamp, the DN report gets
> requeued.
> However, in {{FSNamesystem#storeAllocatedBlock}}, we have the following code:
> {code:java}
> // allocate new block, record block locations in INode.
> newBlock = createNewBlock();
> INodesInPath inodesInPath = INodesInPath.fromINode(pendingFile);
> saveAllocatedBlock(src, inodesInPath, newBlock, targets);
> persistNewBlock(src, pendingFile);
> offset = pendingFile.computeFileSize();
> {code}
> The line
> {{newBlock = createNewBlock();}}
> Would log an edit entry {{OP_SET_GENSTAMP_V2}} to bump generation stamp on
> Standby
> while the following line
> {{persistNewBlock(src, pendingFile);}}
> would log another edit entry {{OP_ADD_BLOCK}} to actually add the block on
> Standby.
> Then the race condition is that, imagine Standby has just processed
> {{OP_SET_GENSTAMP_V2}}, but not yet {{OP_ADD_BLOCK}} (if they just happen to
> be in different setment). Now a block report with new generation stamp comes
> in.
> Since the genstamp bump has already been processed, the reported block may
> not be considered as future block. So the guarding logic passes. But
> actually, the block hasn't been added to blockmap, because the second edit is
> yet to be tailed. So, the block then gets added to invalidate block list and
> we saw messages like:
> {code:java}
> BLOCK* addBlock: block XXX on node XXX size XXX does not belong to any file
> {code}
> Even worse, since this IBR is effectively lost, the NameNode has no
> information about this block, until the next full block report. So after a
> failover, the NN marks it as corrupt.
> This issue won't happen though, if both of the edit entries get tailed all
> together, so no IBR processing can happen in between. But in our case, we set
> edit tailing interval to super low (to allow Standby read), so when under
> high workload, there is a much much higher chance that the two entries are
> tailed separately, causing the issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14941) Potential editlog race condition can cause corrupted file
Chen Liang created HDFS-14941:
-
Summary: Potential editlog race condition can cause corrupted file
Key: HDFS-14941
URL: https://issues.apache.org/jira/browse/HDFS-14941
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Chen Liang
Recently we encountered an issue that, after a failover, NameNode complains
corrupted file/missing blocks. The blocks did recover after full block reports,
so the blocks are not actually missing. After further investigation, we believe
this is what happened:
First of all, on SbN, it is possible that it receives block reports before
corresponding edit tailing happened. In which case SbN postpones processing the
DN block report, handled by the guarding logic below:
{code:java}
if (shouldPostponeBlocksFromFuture &&
namesystem.isGenStampInFuture(iblk)) {
queueReportedBlock(storageInfo, iblk, reportedState,
QUEUE_REASON_FUTURE_GENSTAMP);
continue;
}
{code}
Basically if reported block has a future generation stamp, the DN report gets
requeued.
However, in {{FSNamesystem#storeAllocatedBlock}}, we have the following code:
{code:java}
// allocate new block, record block locations in INode.
newBlock = createNewBlock();
INodesInPath inodesInPath = INodesInPath.fromINode(pendingFile);
saveAllocatedBlock(src, inodesInPath, newBlock, targets);
persistNewBlock(src, pendingFile);
offset = pendingFile.computeFileSize();
{code}
The line
{{newBlock = createNewBlock();}}
Would log an edit entry {{OP_SET_GENSTAMP_V2}} to bump generation stamp on
Standby
while the following line
{{persistNewBlock(src, pendingFile);}}
would log another edit entry {{OP_ADD_BLOCK}} to actually add the block on
Standby.
Then the race condition is that, imagine Standby has just processed
{{OP_SET_GENSTAMP_V2}}, but not yet {{OP_ADD_BLOCK}} (if they just happen to be
in different setment). Now a block report with new generation stamp comes in.
Since the genstamp bump has already been processed, the reported block may not
be considered as future block. So the guarding logic passes. But actually, the
block hasn't been added to blockmap, because the second edit is yet to be
tailed. So, the block then gets added to invalidate block list and we saw
messages like:
{code:java}
BLOCK* addBlock: block XXX on node XXX size XXX does not belong to any file
{code}
Even worse, since this IBR is effectively lost, the NameNode has no information
about this block, until the next full block report. So after a failover, the NN
marks it as corrupt.
This issue won't happen though, if both of the edit entries get tailed all
together, so no IBR processing can happen in between. But in our case, we set
edit tailing interval to super low (to allow Standby read), so when under high
workload, there is a much much higher chance that the two entries are tailed
separately, causing the issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2380) Use the Table.isExist API instead of get() call while checking for presence of key.
[ https://issues.apache.org/jira/browse/HDDS-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-2380: Summary: Use the Table.isExist API instead of get() call while checking for presence of key. (was: OMFileRequest should use the isExist API while checking for pre-existing files in the directory path. ) > Use the Table.isExist API instead of get() call while checking for presence > of key. > --- > > Key: HDDS-2380 > URL: https://issues.apache.org/jira/browse/HDDS-2380 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, when OM creates a file/directory, it checks the absence of all > prefix paths of the key in its RocksDB. Since we don't care about the > deserialization of the actual value, we should use the isExist API added in > org.apache.hadoop.hdds.utils.db.Table which internally uses the more > performant keyMayExist API of RocksDB. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-426) Add field modificationTime for Volume and Bucket
[
https://issues.apache.org/jira/browse/HDDS-426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962528#comment-16962528
]
Arpit Agarwal commented on HDDS-426:
[~aengineer] I updated the resolution. However I realized the two linked bugs
added createTime, this jira is requesting a modifiedTime.
> Add field modificationTime for Volume and Bucket
>
>
> Key: HDDS-426
> URL: https://issues.apache.org/jira/browse/HDDS-426
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Manager
>Reporter: Dinesh Chitlangia
>Assignee: YiSheng Lien
>Priority: Major
> Labels: newbie
>
> There are update operations that can be performed for Volume, Bucket and Key.
> While Key records the modification time, Volume and & Bucket do not capture
> this.
>
> This Jira proposes to add the required field to Volume and Bucket in order to
> capture the modficationTime.
>
> Current Status:
> {noformat}
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoVolume /dummyvol
> 2018-09-10 17:16:12 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "owner" : {
> "name" : "bilbo"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "dummyvol",
> "createdOn" : "Mon, 10 Sep 2018 17:11:32 GMT",
> "createdBy" : "bilbo"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoBucket /dummyvol/mybuck
> 2018-09-10 17:15:25 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "volumeName" : "dummyvol",
> "bucketName" : "mybuck",
> "createdOn" : "Mon, 10 Sep 2018 17:12:09 GMT",
> "acls" : [ {
> "type" : "USER",
> "name" : "hadoop",
> "rights" : "READ_WRITE"
> }, {
> "type" : "GROUP",
> "name" : "users",
> "rights" : "READ_WRITE"
> }, {
> "type" : "USER",
> "name" : "spark",
> "rights" : "READ_WRITE"
> } ],
> "versioning" : "DISABLED",
> "storageType" : "DISK"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoKey /dummyvol/mybuck/myk1
> 2018-09-10 17:19:43 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "modifiedOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "size" : 0,
> "keyName" : "myk1",
> "keyLocations" : [ ]
> }{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-426) Add field modificationTime for Volume and Bucket
[
https://issues.apache.org/jira/browse/HDDS-426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal resolved HDDS-426.
Resolution: Duplicate
> Add field modificationTime for Volume and Bucket
>
>
> Key: HDDS-426
> URL: https://issues.apache.org/jira/browse/HDDS-426
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Manager
>Reporter: Dinesh Chitlangia
>Assignee: YiSheng Lien
>Priority: Major
> Labels: newbie
>
> There are update operations that can be performed for Volume, Bucket and Key.
> While Key records the modification time, Volume and & Bucket do not capture
> this.
>
> This Jira proposes to add the required field to Volume and Bucket in order to
> capture the modficationTime.
>
> Current Status:
> {noformat}
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoVolume /dummyvol
> 2018-09-10 17:16:12 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "owner" : {
> "name" : "bilbo"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "dummyvol",
> "createdOn" : "Mon, 10 Sep 2018 17:11:32 GMT",
> "createdBy" : "bilbo"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoBucket /dummyvol/mybuck
> 2018-09-10 17:15:25 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "volumeName" : "dummyvol",
> "bucketName" : "mybuck",
> "createdOn" : "Mon, 10 Sep 2018 17:12:09 GMT",
> "acls" : [ {
> "type" : "USER",
> "name" : "hadoop",
> "rights" : "READ_WRITE"
> }, {
> "type" : "GROUP",
> "name" : "users",
> "rights" : "READ_WRITE"
> }, {
> "type" : "USER",
> "name" : "spark",
> "rights" : "READ_WRITE"
> } ],
> "versioning" : "DISABLED",
> "storageType" : "DISK"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoKey /dummyvol/mybuck/myk1
> 2018-09-10 17:19:43 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "modifiedOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "size" : 0,
> "keyName" : "myk1",
> "keyLocations" : [ ]
> }{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDDS-426) Add field modificationTime for Volume and Bucket
[
https://issues.apache.org/jira/browse/HDDS-426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal reopened HDDS-426:
> Add field modificationTime for Volume and Bucket
>
>
> Key: HDDS-426
> URL: https://issues.apache.org/jira/browse/HDDS-426
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Manager
>Reporter: Dinesh Chitlangia
>Assignee: YiSheng Lien
>Priority: Major
> Labels: newbie
>
> There are update operations that can be performed for Volume, Bucket and Key.
> While Key records the modification time, Volume and & Bucket do not capture
> this.
>
> This Jira proposes to add the required field to Volume and Bucket in order to
> capture the modficationTime.
>
> Current Status:
> {noformat}
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoVolume /dummyvol
> 2018-09-10 17:16:12 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "owner" : {
> "name" : "bilbo"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "dummyvol",
> "createdOn" : "Mon, 10 Sep 2018 17:11:32 GMT",
> "createdBy" : "bilbo"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoBucket /dummyvol/mybuck
> 2018-09-10 17:15:25 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "volumeName" : "dummyvol",
> "bucketName" : "mybuck",
> "createdOn" : "Mon, 10 Sep 2018 17:12:09 GMT",
> "acls" : [ {
> "type" : "USER",
> "name" : "hadoop",
> "rights" : "READ_WRITE"
> }, {
> "type" : "GROUP",
> "name" : "users",
> "rights" : "READ_WRITE"
> }, {
> "type" : "USER",
> "name" : "spark",
> "rights" : "READ_WRITE"
> } ],
> "versioning" : "DISABLED",
> "storageType" : "DISK"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoKey /dummyvol/mybuck/myk1
> 2018-09-10 17:19:43 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "modifiedOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "size" : 0,
> "keyName" : "myk1",
> "keyLocations" : [ ]
> }{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-426) Add field modificationTime for Volume and Bucket
[
https://issues.apache.org/jira/browse/HDDS-426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-426:
---
Fix Version/s: (was: 0.5.0)
> Add field modificationTime for Volume and Bucket
>
>
> Key: HDDS-426
> URL: https://issues.apache.org/jira/browse/HDDS-426
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Manager
>Reporter: Dinesh Chitlangia
>Assignee: YiSheng Lien
>Priority: Major
> Labels: newbie
>
> There are update operations that can be performed for Volume, Bucket and Key.
> While Key records the modification time, Volume and & Bucket do not capture
> this.
>
> This Jira proposes to add the required field to Volume and Bucket in order to
> capture the modficationTime.
>
> Current Status:
> {noformat}
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoVolume /dummyvol
> 2018-09-10 17:16:12 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "owner" : {
> "name" : "bilbo"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "dummyvol",
> "createdOn" : "Mon, 10 Sep 2018 17:11:32 GMT",
> "createdBy" : "bilbo"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoBucket /dummyvol/mybuck
> 2018-09-10 17:15:25 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "volumeName" : "dummyvol",
> "bucketName" : "mybuck",
> "createdOn" : "Mon, 10 Sep 2018 17:12:09 GMT",
> "acls" : [ {
> "type" : "USER",
> "name" : "hadoop",
> "rights" : "READ_WRITE"
> }, {
> "type" : "GROUP",
> "name" : "users",
> "rights" : "READ_WRITE"
> }, {
> "type" : "USER",
> "name" : "spark",
> "rights" : "READ_WRITE"
> } ],
> "versioning" : "DISABLED",
> "storageType" : "DISK"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoKey /dummyvol/mybuck/myk1
> 2018-09-10 17:19:43 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "modifiedOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "size" : 0,
> "keyName" : "myk1",
> "keyLocations" : [ ]
> }{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2376) Fail to read data through XceiverClientGrpc
[ https://issues.apache.org/jira/browse/HDDS-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey reassigned HDDS-2376: -- Assignee: Hanisha Koneru > Fail to read data through XceiverClientGrpc > --- > > Key: HDDS-2376 > URL: https://issues.apache.org/jira/browse/HDDS-2376 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Sammi Chen >Assignee: Hanisha Koneru >Priority: Blocker > > Run teragen, application failed with following stack, > 19/10/29 14:35:42 INFO mapreduce.Job: Running job: job_1567133159094_0048 > 19/10/29 14:35:59 INFO mapreduce.Job: Job job_1567133159094_0048 running in > uber mode : false > 19/10/29 14:35:59 INFO mapreduce.Job: map 0% reduce 0% > 19/10/29 14:35:59 INFO mapreduce.Job: Job job_1567133159094_0048 failed with > state FAILED due to: Application application_1567133159094_0048 failed 2 > times due to AM Container for appattempt_1567133159094_0048_02 exited > with exitCode: -1000 > For more detailed output, check application tracking > page:http://host183:8088/cluster/app/application_1567133159094_0048Then, > click on links to logs of each attempt. > Diagnostics: Unexpected OzoneException: > org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum mismatch at > index 0 > java.io.IOException: Unexpected OzoneException: > org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum mismatch at > index 0 > at > org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:342) > at > org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:307) > at > org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:259) > at > org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:144) > at > org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:239) > at > org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171) > at > org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:52) > at java.io.DataInputStream.read(DataInputStream.java:100) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:86) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:60) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:120) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366) > at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:267) > at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63) > at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361) > at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754) > at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:359) > at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum > mismatch at index 0 > at > org.apache.hadoop.ozone.common.ChecksumData.verifyChecksumDataMatches(ChecksumData.java:148) > at > org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:275) > at > org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:238) > at > org.apache.hadoop.hdds.scm.storage.ChunkInputStream.lambda$new$0(ChunkInputStream.java:375) > at > org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:287) > at > org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:250) > at > org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:233) > at > org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:245) > at > org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:335) > ... 26 more > Caused by: Checksum mismatch at index 0 > org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum mismatch at > index 0 >
[jira] [Work logged] (HDDS-2381) In ExcludeList, add if not exist only
[ https://issues.apache.org/jira/browse/HDDS-2381?focusedWorklogId=335813=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-335813 ] ASF GitHub Bot logged work on HDDS-2381: Author: ASF GitHub Bot Created on: 29/Oct/19 23:05 Start Date: 29/Oct/19 23:05 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #104: HDDS-2381. In ExcludeList, add if not exist only. URL: https://github.com/apache/hadoop-ozone/pull/104 ## What changes were proposed in this pull request? Add a check if not contains for addDatanode, addPipeline, addContainer. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2381 ## How was this patch tested? The check added will take care, and the log will not print same pipeline. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 335813) Remaining Estimate: 0h Time Spent: 10m > In ExcludeList, add if not exist only > - > > Key: HDDS-2381 > URL: https://issues.apache.org/jira/browse/HDDS-2381 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Created based on comment from [~chinseone] in HDDS-2356 > https://issues.apache.org/jira/browse/HDDS-2356?focusedCommentId=16960796=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16960796 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2381) In ExcludeList, add if not exist only
[ https://issues.apache.org/jira/browse/HDDS-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2381: - Labels: pull-request-available (was: ) > In ExcludeList, add if not exist only > - > > Key: HDDS-2381 > URL: https://issues.apache.org/jira/browse/HDDS-2381 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > Created based on comment from [~chinseone] in HDDS-2356 > https://issues.apache.org/jira/browse/HDDS-2356?focusedCommentId=16960796=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16960796 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-426) Add field modificationTime for Volume and Bucket
[
https://issues.apache.org/jira/browse/HDDS-426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer resolved HDDS-426.
---
Fix Version/s: 0.5.0
Resolution: Fixed
Looks like HDDS-1551 added Creation Time to bucketInfo and HDDS-1620 added
creationTime to VolumeInfo.
> Add field modificationTime for Volume and Bucket
>
>
> Key: HDDS-426
> URL: https://issues.apache.org/jira/browse/HDDS-426
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Manager
>Reporter: Dinesh Chitlangia
>Assignee: YiSheng Lien
>Priority: Major
> Labels: newbie
> Fix For: 0.5.0
>
>
> There are update operations that can be performed for Volume, Bucket and Key.
> While Key records the modification time, Volume and & Bucket do not capture
> this.
>
> This Jira proposes to add the required field to Volume and Bucket in order to
> capture the modficationTime.
>
> Current Status:
> {noformat}
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoVolume /dummyvol
> 2018-09-10 17:16:12 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "owner" : {
> "name" : "bilbo"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "dummyvol",
> "createdOn" : "Mon, 10 Sep 2018 17:11:32 GMT",
> "createdBy" : "bilbo"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoBucket /dummyvol/mybuck
> 2018-09-10 17:15:25 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "volumeName" : "dummyvol",
> "bucketName" : "mybuck",
> "createdOn" : "Mon, 10 Sep 2018 17:12:09 GMT",
> "acls" : [ {
> "type" : "USER",
> "name" : "hadoop",
> "rights" : "READ_WRITE"
> }, {
> "type" : "GROUP",
> "name" : "users",
> "rights" : "READ_WRITE"
> }, {
> "type" : "USER",
> "name" : "spark",
> "rights" : "READ_WRITE"
> } ],
> "versioning" : "DISABLED",
> "storageType" : "DISK"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoKey /dummyvol/mybuck/myk1
> 2018-09-10 17:19:43 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "modifiedOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "size" : 0,
> "keyName" : "myk1",
> "keyLocations" : [ ]
> }{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2381) In ExcludeList, add if not exist only
[ https://issues.apache.org/jira/browse/HDDS-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-2381: - Issue Type: Bug (was: Task) > In ExcludeList, add if not exist only > - > > Key: HDDS-2381 > URL: https://issues.apache.org/jira/browse/HDDS-2381 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > Created based on comment from [~chinseone] in HDDS-2356 > https://issues.apache.org/jira/browse/HDDS-2356?focusedCommentId=16960796=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16960796 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2381) In ExcludeList, add if not exist only
Bharat Viswanadham created HDDS-2381: Summary: In ExcludeList, add if not exist only Key: HDDS-2381 URL: https://issues.apache.org/jira/browse/HDDS-2381 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham Created based on comment from [~chinseone] in HDDS-2356 https://issues.apache.org/jira/browse/HDDS-2356?focusedCommentId=16960796=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16960796 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14927) RBF: Add metrics for async callers thread pool
[ https://issues.apache.org/jira/browse/HDFS-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962503#comment-16962503 ] Íñigo Goiri commented on HDFS-14927: So we are relying on the delay added when doing the slow DN. Not the best but I guess we can leave with that. Regarding the sleep, let's remove it and replace it with a GenericTestUtils#waitFor(). > RBF: Add metrics for async callers thread pool > -- > > Key: HDFS-14927 > URL: https://issues.apache.org/jira/browse/HDFS-14927 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Minor > Attachments: HDFS-14927.001.patch, HDFS-14927.002.patch, > HDFS-14927.003.patch, HDFS-14927.004.patch, HDFS-14927.005.patch > > > It is good to add some monitoring on the async caller thread pool to handle > fan-out RPC client requests, so we know the utilization and when to bump up > dfs.federation.router.client.thread-size -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2322) DoubleBuffer flush termination and OM shutdown's after that.
[ https://issues.apache.org/jira/browse/HDDS-2322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-2322: - Fix Version/s: 0.5.0 Resolution: Fixed Status: Resolved (was: Patch Available) > DoubleBuffer flush termination and OM shutdown's after that. > > > Key: HDDS-2322 > URL: https://issues.apache.org/jira/browse/HDDS-2322 > Project: Hadoop Distributed Data Store > Issue Type: Task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > om1_1 | 2019-10-18 00:34:45,317 [OMDoubleBufferFlushThread] ERROR > - Terminating with exit status 2: OMDoubleBuffer flush > threadOMDoubleBufferFlushThreadencountered Throwable error > om1_1 | java.util.ConcurrentModificationException > om1_1 | at > java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1660) > om1_1 | at > java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) > om1_1 | at > java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) > om1_1 | at > java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) > om1_1 | at > java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > om1_1 | at > java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) > om1_1 | at > org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup.getProtobuf(OmKeyLocationInfoGroup.java:65) > om1_1 | at > java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) > om1_1 | at > java.base/java.util.Collections$2.tryAdvance(Collections.java:4745) > om1_1 | at > java.base/java.util.Collections$2.forEachRemaining(Collections.java:4753) > om1_1 | at > java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) > om1_1 | at > java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) > om1_1 | at > java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) > om1_1 | at > java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > om1_1 | at > java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) > om1_1 | at > org.apache.hadoop.ozone.om.helpers.OmKeyInfo.getProtobuf(OmKeyInfo.java:362) > om1_1 | at > org.apache.hadoop.ozone.om.codec.OmKeyInfoCodec.toPersistedFormat(OmKeyInfoCodec.java:37) > om1_1 | at > org.apache.hadoop.ozone.om.codec.OmKeyInfoCodec.toPersistedFormat(OmKeyInfoCodec.java:31) > om1_1 | at > org.apache.hadoop.hdds.utils.db.CodecRegistry.asRawData(CodecRegistry.java:68) > om1_1 | at > org.apache.hadoop.hdds.utils.db.TypedTable.putWithBatch(TypedTable.java:125) > om1_1 | at > org.apache.hadoop.ozone.om.response.key.OMKeyCreateResponse.addToDBBatch(OMKeyCreateResponse.java:58) > om1_1 | at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$flushTransactions$0(OzoneManagerDoubleBuffer.java:139) > om1_1 | at > java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) > om1_1 | at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:137) > om1_1 | at java.base/java.lang.Thread.run(Thread.java:834) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2322) DoubleBuffer flush termination and OM shutdown's after that.
[ https://issues.apache.org/jira/browse/HDDS-2322?focusedWorklogId=335796=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-335796 ] ASF GitHub Bot logged work on HDDS-2322: Author: ASF GitHub Bot Created on: 29/Oct/19 22:27 Start Date: 29/Oct/19 22:27 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #89: HDDS-2322. DoubleBuffer flush termination and OM shutdown's after that. Make entry returned from cache a new copy. URL: https://github.com/apache/hadoop-ozone/pull/89 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 335796) Time Spent: 20m (was: 10m) > DoubleBuffer flush termination and OM shutdown's after that. > > > Key: HDDS-2322 > URL: https://issues.apache.org/jira/browse/HDDS-2322 > Project: Hadoop Distributed Data Store > Issue Type: Task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > om1_1 | 2019-10-18 00:34:45,317 [OMDoubleBufferFlushThread] ERROR > - Terminating with exit status 2: OMDoubleBuffer flush > threadOMDoubleBufferFlushThreadencountered Throwable error > om1_1 | java.util.ConcurrentModificationException > om1_1 | at > java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1660) > om1_1 | at > java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) > om1_1 | at > java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) > om1_1 | at > java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) > om1_1 | at > java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > om1_1 | at > java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) > om1_1 | at > org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup.getProtobuf(OmKeyLocationInfoGroup.java:65) > om1_1 | at > java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) > om1_1 | at > java.base/java.util.Collections$2.tryAdvance(Collections.java:4745) > om1_1 | at > java.base/java.util.Collections$2.forEachRemaining(Collections.java:4753) > om1_1 | at > java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) > om1_1 | at > java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) > om1_1 | at > java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) > om1_1 | at > java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > om1_1 | at > java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) > om1_1 | at > org.apache.hadoop.ozone.om.helpers.OmKeyInfo.getProtobuf(OmKeyInfo.java:362) > om1_1 | at > org.apache.hadoop.ozone.om.codec.OmKeyInfoCodec.toPersistedFormat(OmKeyInfoCodec.java:37) > om1_1 | at > org.apache.hadoop.ozone.om.codec.OmKeyInfoCodec.toPersistedFormat(OmKeyInfoCodec.java:31) > om1_1 | at > org.apache.hadoop.hdds.utils.db.CodecRegistry.asRawData(CodecRegistry.java:68) > om1_1 | at > org.apache.hadoop.hdds.utils.db.TypedTable.putWithBatch(TypedTable.java:125) > om1_1 | at > org.apache.hadoop.ozone.om.response.key.OMKeyCreateResponse.addToDBBatch(OMKeyCreateResponse.java:58) > om1_1 | at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$flushTransactions$0(OzoneManagerDoubleBuffer.java:139) > om1_1 | at > java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) > om1_1 | at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:137) > om1_1 | at java.base/java.lang.Thread.run(Thread.java:834) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2292) Create Ozone specific README.md to the new hadoop-ozone repository
[ https://issues.apache.org/jira/browse/HDDS-2292?focusedWorklogId=335793=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-335793 ] ASF GitHub Bot logged work on HDDS-2292: Author: ASF GitHub Bot Created on: 29/Oct/19 22:14 Start Date: 29/Oct/19 22:14 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #55: HDDS-2292. Create Ozone specific README.md to the new hadoop-ozone re… URL: https://github.com/apache/hadoop-ozone/pull/55 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 335793) Time Spent: 20m (was: 10m) > Create Ozone specific README.md to the new hadoop-ozone repository > -- > > Key: HDDS-2292 > URL: https://issues.apache.org/jira/browse/HDDS-2292 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The current README is main Hadoop specific. We can create an ozone specific. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14927) RBF: Add metrics for async callers thread pool
[
https://issues.apache.org/jira/browse/HDFS-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962488#comment-16962488
]
Leon Gao commented on HDFS-14927:
-
Ah I got your question.
How it works is that TestRouterClientRejectOverload#375:
{code:java}
simulateSlowNamenode(cluster.getCluster().getNameNode(0), 2);{code}
Above makes sure that renewLease() call will take at least 2 seconds to finish.
This means the async thread will keep the connection working for at least 2
seconds in background.
The metrics validation happens in 0.5 seconds, so it will for sure drop in the
2 seconds interval and get one active connection.
The reason to sleep for 0.5 seconds is to make sure the connection is already
active in the connection pool, which might just take some milliseconds on my
laptop. The wait makes sure the test is less likely to be flaky :)
I was trying sync call at the beginning which doesn't seem to be working well..
Maybe the time connection stayed afterward was too short to make the test
consistent.
> RBF: Add metrics for async callers thread pool
> --
>
> Key: HDFS-14927
> URL: https://issues.apache.org/jira/browse/HDFS-14927
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Leon Gao
>Assignee: Leon Gao
>Priority: Minor
> Attachments: HDFS-14927.001.patch, HDFS-14927.002.patch,
> HDFS-14927.003.patch, HDFS-14927.004.patch, HDFS-14927.005.patch
>
>
> It is good to add some monitoring on the async caller thread pool to handle
> fan-out RPC client requests, so we know the utilization and when to bump up
> dfs.federation.router.client.thread-size
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2378) Change "OZONE" as string used in the code where OzoneConsts.OZONE is suitable
[
https://issues.apache.org/jira/browse/HDDS-2378?focusedWorklogId=335779=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-335779
]
ASF GitHub Bot logged work on HDDS-2378:
Author: ASF GitHub Bot
Created on: 29/Oct/19 21:36
Start Date: 29/Oct/19 21:36
Worklog Time Spent: 10m
Work Description: fapifta commented on pull request #103: HDDS-2378 -
Change OZONE as string used in the code where OzoneConsts.OZONE is suitable
URL: https://github.com/apache/hadoop-ozone/pull/103
## What changes were proposed in this pull request?
In #95 review it came up that the tests could use the OzoneConsts.OZONE
constant instead of using the "ozone" string literal, this PR is aiming to
address this issue globally in the whole codebase.
As it is commented in the JIRA the changes were made in this PR where
suitable in the following way:
There are some places where we use the string literal "Ozone" which I would
not change, the places are:
- RocksDBStore, and RDBStore, where it is used to specify the name of jmx
keys.
- ServerUtils uses it in a function parameter passed to a log message that
seems fine.
- StorageContainerManager, and OzoneManager uses the full capital in
InterfaceAudience annotation, that is fine.
- CommonHeadersContainerResponseFilter is using this in headers I would not
change the protocol
In TestAuthorizationHeaderV4 we use "ozone" that is ok, as we provide a
string to parse, and check elements provided, in this case I would not change
to a constant makes it harder to read.
CreateSubcommand I am unsure, the default is specified with full capital I
am not brave enough to change it to the lowercase version, though my instinct
tells that it would be most likely ok.
Also while going through the S3 related tests, that were coming up with the
query, I changed a few other literals to suitable constants, like:
"OWNER" -> OzoneConsts.OWNER (makes it lowercase)
"VOLUME" -> OzoneConsts.VOLUME (makes it lowercase)
"hdfs" -> OzoneConsts.OZONE_SIMPLE_HDFS_USER
"xxBucket" -> OzoneConsts.BUCKET
"s3Bucket" -> OzoneConsts.S3_BUCKET
"key1" -> OzoneConsts.KEY (in cases where only 1 key is used)
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-2378
## How was this patch tested?
There are no change in the code logic, mostly tests were changed, the
expectation is that all the JUnit tests should run as they ran before. I have
ran most of the changed tests locally, they seemed ok.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 335779)
Remaining Estimate: 0h
Time Spent: 10m
> Change "OZONE" as string used in the code where OzoneConsts.OZONE is suitable
> -
>
> Key: HDDS-2378
> URL: https://issues.apache.org/jira/browse/HDDS-2378
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.4.1
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Based on a review I have done a quick check, and there are quite a few places
> where we have hardcoded "ozone" as String literal or a capital version of it
> into the code.
> Let's check then one by one, and where it is possible replace it with
> OzoneConsts.OZONE, or if the lower case version is not acceptable at all
> places, then create an other constant with the uppercase version and use that.
> This is the search, and the results:
> {code:bash}
> find . -name *.java | while read FILE; do NUM=`grep -c -i "\"OZONE\"" $FILE`;
> if [ $NUM -gt 0 ]; then echo $FILE; fi; done | sort | uniq
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDataYaml.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestBlockManagerImpl.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java
>
[jira] [Updated] (HDDS-2378) Change "OZONE" as string used in the code where OzoneConsts.OZONE is suitable
[
https://issues.apache.org/jira/browse/HDDS-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-2378:
-
Labels: pull-request-available (was: )
> Change "OZONE" as string used in the code where OzoneConsts.OZONE is suitable
> -
>
> Key: HDDS-2378
> URL: https://issues.apache.org/jira/browse/HDDS-2378
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.4.1
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
> Labels: pull-request-available
>
> Based on a review I have done a quick check, and there are quite a few places
> where we have hardcoded "ozone" as String literal or a capital version of it
> into the code.
> Let's check then one by one, and where it is possible replace it with
> OzoneConsts.OZONE, or if the lower case version is not acceptable at all
> places, then create an other constant with the uppercase version and use that.
> This is the search, and the results:
> {code:bash}
> find . -name *.java | while read FILE; do NUM=`grep -c -i "\"OZONE\"" $FILE`;
> if [ $NUM -gt 0 ]; then echo $FILE; fi; done | sort | uniq
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDataYaml.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestBlockManagerImpl.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java
> ./hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ServerUtils.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/SCMContainerManagerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/block/TestBlockManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestCloseContainerEventHandler.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSCMContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestContainerPlacement.java
> ./hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/CreateSubcommand.java
> ./hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/OzoneVersionInfo.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/metrics/TestSCMContainerManagerMetrics.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerOperations.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerStateMachineIdempotency.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestS3Shell.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestAllocateContainer.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSmallFile.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestGetCommittedBlockLengthAndPutKey.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientMetrics.java
>
[jira] [Commented] (HDDS-2378) Change "OZONE" as string used in the code where OzoneConsts.OZONE is suitable
[
https://issues.apache.org/jira/browse/HDDS-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962479#comment-16962479
]
Istvan Fajth commented on HDDS-2378:
After going through the code, the following came out of this:
There are some places where we use the string literal "Ozone" which I would not
change, the places are:
- RocksDBStore, and RDBStore, where it is used to specify the name of jmx keys.
- ServerUtils uses it in a function parameter passed to a log message that
seems fine.
- StorageContainerManager, and OzoneManager uses the full capital in
InterfaceAudience annotation, that is fine.
- CommonHeadersContainerResponseFilter is using this in headers I would not
change the protocol
In TestAuthorizationHeaderV4 we use "ozone" that is ok, as we provide a string
to parse, and check elements provided, in this case I would not change to a
constant makes it harder to read.
CreateSubcommand I am unsure, the default is specified with full capital I am
not brave enough to change it to the lowercase version, though my instinct
tells that it would be most likely ok.
Also while going through the S3 related tests, that were coming up with the
query, I changed a few other literals to suitable constants, like:
"OWNER" -> OzoneConsts.OWNER (makes it lowercase)
"VOLUME" -> OzoneConsts.VOLUME (makes it lowercase)
"hdfs" -> OzoneConsts.OZONE_SIMPLE_HDFS_USER
"xxBucket" -> OzoneConsts.BUCKET
"s3Bucket" -> OzoneConsts.S3_BUCKET
"key1" -> OzoneConsts.KEY (in cases where only 1 key is used)
Preparing the PR shortly.
> Change "OZONE" as string used in the code where OzoneConsts.OZONE is suitable
> -
>
> Key: HDDS-2378
> URL: https://issues.apache.org/jira/browse/HDDS-2378
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.4.1
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> Based on a review I have done a quick check, and there are quite a few places
> where we have hardcoded "ozone" as String literal or a capital version of it
> into the code.
> Let's check then one by one, and where it is possible replace it with
> OzoneConsts.OZONE, or if the lower case version is not acceptable at all
> places, then create an other constant with the uppercase version and use that.
> This is the search, and the results:
> {code:bash}
> find . -name *.java | while read FILE; do NUM=`grep -c -i "\"OZONE\"" $FILE`;
> if [ $NUM -gt 0 ]; then echo $FILE; fi; done | sort | uniq
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDataYaml.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestBlockManagerImpl.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java
> ./hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ServerUtils.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/SCMContainerManagerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/block/TestBlockManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestCloseContainerEventHandler.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSCMContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestContainerPlacement.java
> ./hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/CreateSubcommand.java
> ./hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/OzoneVersionInfo.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/metrics/TestSCMContainerManagerMetrics.java
>
[jira] [Work logged] (HDDS-2380) OMFileRequest should use the isExist API while checking for pre-existing files in the directory path.
[ https://issues.apache.org/jira/browse/HDDS-2380?focusedWorklogId=335752=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-335752 ] ASF GitHub Bot logged work on HDDS-2380: Author: ASF GitHub Bot Created on: 29/Oct/19 21:00 Start Date: 29/Oct/19 21:00 Worklog Time Spent: 10m Work Description: avijayanhwx commented on pull request #102: HDDS-2380. OMFileRequest should use the isExist API while checking fo… URL: https://github.com/apache/hadoop-ozone/pull/102 …r pre-existing files in the directory path. ## What changes were proposed in this pull request? Currently, when OM creates a file/directory, it checks the absence of all prefix paths of the key in its RocksDB. Since we don't care about the deserialization of the actual value from the DB, we should use the isExist API added in org.apache.hadoop.hdds.utils.db.Table which internally uses the more performant keyMayExist API of RocksDB. More documentation on the keyMayExist API can be found [here](https://github.com/facebook/rocksdb/blob/7a8d7358bb40b13a06c2c6adc62e80295d89ed05/java/src/main/java/org/rocksdb/RocksDB.java#L2184) This API was already used in the old OM create file flow. This patch adds it to the new OM code flow that uses Double buffering. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2380 ## How was this patch tested? Manually tested. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 335752) Remaining Estimate: 0h Time Spent: 10m > OMFileRequest should use the isExist API while checking for pre-existing > files in the directory path. > -- > > Key: HDDS-2380 > URL: https://issues.apache.org/jira/browse/HDDS-2380 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, when OM creates a file/directory, it checks the absence of all > prefix paths of the key in its RocksDB. Since we don't care about the > deserialization of the actual value, we should use the isExist API added in > org.apache.hadoop.hdds.utils.db.Table which internally uses the more > performant keyMayExist API of RocksDB. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2380) OMFileRequest should use the isExist API while checking for pre-existing files in the directory path.
[ https://issues.apache.org/jira/browse/HDDS-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2380: - Labels: pull-request-available (was: ) > OMFileRequest should use the isExist API while checking for pre-existing > files in the directory path. > -- > > Key: HDDS-2380 > URL: https://issues.apache.org/jira/browse/HDDS-2380 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > > Currently, when OM creates a file/directory, it checks the absence of all > prefix paths of the key in its RocksDB. Since we don't care about the > deserialization of the actual value, we should use the isExist API added in > org.apache.hadoop.hdds.utils.db.Table which internally uses the more > performant keyMayExist API of RocksDB. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13507) RBF: Remove update functionality from routeradmin's add cmd
[
https://issues.apache.org/jira/browse/HDFS-13507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962464#comment-16962464
]
Hadoop QA commented on HDFS-13507:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 44s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
10s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHDFS |
| | hadoop.hdfs.TestMultipleNNPortQOP |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-13507 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12984296/HDFS-13507.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 80368f3d70ee 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bd498ba |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (HDFS-14927) RBF: Add metrics for async callers thread pool
[ https://issues.apache.org/jira/browse/HDFS-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962457#comment-16962457 ] Íñigo Goiri commented on HDFS-14927: I think I'm missing how this call of renewLease() is done in the background. >From what I can understand here, this will trigger a call in the background >but then wait half a second. By the time the sleep finishes, the call to renewLease() should already be finished. Making it synchronous would be the same and you would still have an active one because the Router leaves the threads on for some time. > RBF: Add metrics for async callers thread pool > -- > > Key: HDFS-14927 > URL: https://issues.apache.org/jira/browse/HDFS-14927 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Minor > Attachments: HDFS-14927.001.patch, HDFS-14927.002.patch, > HDFS-14927.003.patch, HDFS-14927.004.patch, HDFS-14927.005.patch > > > It is good to add some monitoring on the async caller thread pool to handle > fan-out RPC client requests, so we know the utilization and when to bump up > dfs.federation.router.client.thread-size -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2364) Add a OM metrics to find the false positive rate for the keyMayExist
[ https://issues.apache.org/jira/browse/HDDS-2364?focusedWorklogId=335739=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-335739 ] ASF GitHub Bot logged work on HDDS-2364: Author: ASF GitHub Bot Created on: 29/Oct/19 20:42 Start Date: 29/Oct/19 20:42 Worklog Time Spent: 10m Work Description: avijayanhwx commented on pull request #101: HDDS-2364. Add a OM metrics to find the false positive rate for the keyMayExist. URL: https://github.com/apache/hadoop-ozone/pull/101 ## What changes were proposed in this pull request? Added metrics for tracking the RocksDB keyMayExist miss rate. The keyMayExist API usage was added in HDDS-1691. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2364 ## How was this patch tested? Unit tested. Tested manually on cluster by looking at OzoneManager JMX endpoint. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 335739) Remaining Estimate: 0h Time Spent: 10m > Add a OM metrics to find the false positive rate for the keyMayExist > > > Key: HDDS-2364 > URL: https://issues.apache.org/jira/browse/HDDS-2364 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.5.0 >Reporter: Mukul Kumar Singh >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Add a OM metrics to find the false positive rate for the keyMayExist. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2364) Add a OM metrics to find the false positive rate for the keyMayExist
[ https://issues.apache.org/jira/browse/HDDS-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2364: - Labels: pull-request-available (was: ) > Add a OM metrics to find the false positive rate for the keyMayExist > > > Key: HDDS-2364 > URL: https://issues.apache.org/jira/browse/HDDS-2364 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.5.0 >Reporter: Mukul Kumar Singh >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > > Add a OM metrics to find the false positive rate for the keyMayExist. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2355) Om double buffer flush termination with rocksdb error
[ https://issues.apache.org/jira/browse/HDDS-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962447#comment-16962447 ] Aravindan Vijayan commented on HDDS-2355: - No longer to reproduce this after applying the fix in HDDS-2379. > Om double buffer flush termination with rocksdb error > - > > Key: HDDS-2355 > URL: https://issues.apache.org/jira/browse/HDDS-2355 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Aravindan Vijayan >Priority: Blocker > Fix For: 0.5.0 > > > om_1 |java.io.IOException: Unable to write the batch. > om_1 | at > [org.apache.hadoop.hdds.utils.db.RDBBatchOperation.commit(RDBBatchOperation.java:48|http://org.apache.hadoop.hdds.utils.db.rdbbatchoperation.commit%28rdbbatchoperation.java:48/]) > om_1 | at > [org.apache.hadoop.hdds.utils.db.RDBStore.commitBatchOperation(RDBStore.java:240|http://org.apache.hadoop.hdds.utils.db.rdbstore.commitbatchoperation%28rdbstore.java:240/]) > om_1 |at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:146) > om_1 |at java.base/java.lang.Thread.run(Thread.java:834) > om_1 |Caused by: org.rocksdb.RocksDBException: > WritePrepared/WriteUnprepared txn tag when write_after_commit_ is enabled (in > default WriteCommitted mode). If it is not due to corruption, the WAL must be > emptied before changing the WritePolicy. > om_1 |at org.rocksdb.RocksDB.write0(Native Method) > om_1 |at org.rocksdb.RocksDB.write(RocksDB.java:1421) > om_1 | at > [org.apache.hadoop.hdds.utils.db.RDBBatchOperation.commit(RDBBatchOperation.java:46|http://org.apache.hadoop.hdds.utils.db.rdbbatchoperation.commit%28rdbbatchoperation.java:46/]) > > In few of my test run's i see this error and OM is terminated. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2240) Command line tool for OM Admin
[ https://issues.apache.org/jira/browse/HDDS-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-2240: - Fix Version/s: 0.5.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Command line tool for OM Admin > -- > > Key: HDDS-2240 > URL: https://issues.apache.org/jira/browse/HDDS-2240 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > A command line tool (*ozone omha*) to get information related to OM HA. > This Jira proposes to add the _getServiceState_ option for OM HA which lists > all the OMs in the service and their corresponding Ratis server roles > (LEADER/ FOLLOWER). > We can later add more options to this tool. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2240) Command line tool for OM Admin
[ https://issues.apache.org/jira/browse/HDDS-2240?focusedWorklogId=335705=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-335705 ] ASF GitHub Bot logged work on HDDS-2240: Author: ASF GitHub Bot Created on: 29/Oct/19 19:40 Start Date: 29/Oct/19 19:40 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #9: HDDS-2240. Command line tool for OM Admin URL: https://github.com/apache/hadoop-ozone/pull/9 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 335705) Time Spent: 3.5h (was: 3h 20m) > Command line tool for OM Admin > -- > > Key: HDDS-2240 > URL: https://issues.apache.org/jira/browse/HDDS-2240 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > A command line tool (*ozone omha*) to get information related to OM HA. > This Jira proposes to add the _getServiceState_ option for OM HA which lists > all the OMs in the service and their corresponding Ratis server roles > (LEADER/ FOLLOWER). > We can later add more options to this tool. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2378) Change "OZONE" as string used in the code where OzoneConsts.OZONE is suitable
[
https://issues.apache.org/jira/browse/HDDS-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Istvan Fajth updated HDDS-2378:
---
Summary: Change "OZONE" as string used in the code where OzoneConsts.OZONE
is suitable (was: Remove "OZONE" as string used in tests)
> Change "OZONE" as string used in the code where OzoneConsts.OZONE is suitable
> -
>
> Key: HDDS-2378
> URL: https://issues.apache.org/jira/browse/HDDS-2378
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.4.1
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> Based on a review I have done a quick check, and there are quite a few places
> where we have hardcoded "ozone" as String literal or a capital version of it
> into the code.
> Let's check then one by one, and where it is possible replace it with
> OzoneConsts.OZONE, or if the lower case version is not acceptable at all
> places, then create an other constant with the uppercase version and use that.
> This is the search, and the results:
> {code:bash}
> find . -name *.java | while read FILE; do NUM=`grep -c -i "\"OZONE\"" $FILE`;
> if [ $NUM -gt 0 ]; then echo $FILE; fi; done | sort | uniq
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDataYaml.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestBlockManagerImpl.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java
> ./hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ServerUtils.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/SCMContainerManagerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/block/TestBlockManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestCloseContainerEventHandler.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSCMContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestContainerPlacement.java
> ./hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/CreateSubcommand.java
> ./hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/OzoneVersionInfo.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/metrics/TestSCMContainerManagerMetrics.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerOperations.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerStateMachineIdempotency.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestS3Shell.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestAllocateContainer.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSmallFile.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestGetCommittedBlockLengthAndPutKey.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientManager.java
>
[jira] [Created] (HDDS-2380) OMFileRequest should use the isExist API while checking for pre-existing files in the directory path.
Aravindan Vijayan created HDDS-2380: --- Summary: OMFileRequest should use the isExist API while checking for pre-existing files in the directory path. Key: HDDS-2380 URL: https://issues.apache.org/jira/browse/HDDS-2380 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Manager Reporter: Aravindan Vijayan Fix For: 0.5.0 Currently, when OM creates a file/directory, it checks the absence of all prefix paths of the key in its RocksDB. Since we don't care about the deserialization of the actual value, we should use the isExist API added in org.apache.hadoop.hdds.utils.db.Table which internally uses the more performant keyMayExist API of RocksDB. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2380) OMFileRequest should use the isExist API while checking for pre-existing files in the directory path.
[ https://issues.apache.org/jira/browse/HDDS-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-2380: --- Assignee: Aravindan Vijayan > OMFileRequest should use the isExist API while checking for pre-existing > files in the directory path. > -- > > Key: HDDS-2380 > URL: https://issues.apache.org/jira/browse/HDDS-2380 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Fix For: 0.5.0 > > > Currently, when OM creates a file/directory, it checks the absence of all > prefix paths of the key in its RocksDB. Since we don't care about the > deserialization of the actual value, we should use the isExist API added in > org.apache.hadoop.hdds.utils.db.Table which internally uses the more > performant keyMayExist API of RocksDB. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2379) OM terminates with RocksDB error while continuously writing keys.
[
https://issues.apache.org/jira/browse/HDDS-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-2379:
-
Labels: pull-request-available (was: )
> OM terminates with RocksDB error while continuously writing keys.
> -
>
> Key: HDDS-2379
> URL: https://issues.apache.org/jira/browse/HDDS-2379
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Bharat Viswanadham
>Priority: Critical
> Labels: pull-request-available
> Fix For: 0.5.0
>
>
> Exception trace after writing around 800,000 keys.
> {code}
> 2019-10-29 11:15:15,131 ERROR
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer: Terminating with
> exit status 1: During flush to DB encountered err
> or in OMDoubleBuffer flush thread OMDoubleBufferFlushThread
> java.io.IOException: Unable to write the batch.
> at
> org.apache.hadoop.hdds.utils.db.RDBBatchOperation.commit(RDBBatchOperation.java:48)
> at
> org.apache.hadoop.hdds.utils.db.RDBStore.commitBatchOperation(RDBStore.java:240)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:146)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.rocksdb.RocksDBException: unknown WriteBatch tag
> at org.rocksdb.RocksDB.write0(Native Method)
> at org.rocksdb.RocksDB.write(RocksDB.java:1421)
> at
> org.apache.hadoop.hdds.utils.db.RDBBatchOperation.commit(RDBBatchOperation.java:46)
> ... 3 more
> {code}
> Assigning to [~bharat] since he has already started work on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2379) OM terminates with RocksDB error while continuously writing keys.
[
https://issues.apache.org/jira/browse/HDDS-2379?focusedWorklogId=335686=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-335686
]
ASF GitHub Bot logged work on HDDS-2379:
Author: ASF GitHub Bot
Created on: 29/Oct/19 18:50
Start Date: 29/Oct/19 18:50
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #100:
HDDS-2379. OM terminates with RocksDB error while continuously writing keys.
URL: https://github.com/apache/hadoop-ozone/pull/100
## What changes were proposed in this pull request?
Close the BatchOperation, by using try with the resource. As this is not
cleaning up the underlying native references, this occurs randomly.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-2379
Please replace this section with the link to the Apache JIRA)
Ran a few of Integration tests and also after this patch on a cluster
testing we are not seeing this error.
And also the below rocksdb issue link has the same error, once after
following memory management the issue was resolved is the solution proposed.
Followed the same to resolve this issue.
https://github.com/facebook/rocksdb/issues/5068
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 335686)
Remaining Estimate: 0h
Time Spent: 10m
> OM terminates with RocksDB error while continuously writing keys.
> -
>
> Key: HDDS-2379
> URL: https://issues.apache.org/jira/browse/HDDS-2379
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Bharat Viswanadham
>Priority: Critical
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Exception trace after writing around 800,000 keys.
> {code}
> 2019-10-29 11:15:15,131 ERROR
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer: Terminating with
> exit status 1: During flush to DB encountered err
> or in OMDoubleBuffer flush thread OMDoubleBufferFlushThread
> java.io.IOException: Unable to write the batch.
> at
> org.apache.hadoop.hdds.utils.db.RDBBatchOperation.commit(RDBBatchOperation.java:48)
> at
> org.apache.hadoop.hdds.utils.db.RDBStore.commitBatchOperation(RDBStore.java:240)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:146)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.rocksdb.RocksDBException: unknown WriteBatch tag
> at org.rocksdb.RocksDB.write0(Native Method)
> at org.rocksdb.RocksDB.write(RocksDB.java:1421)
> at
> org.apache.hadoop.hdds.utils.db.RDBBatchOperation.commit(RDBBatchOperation.java:46)
> ... 3 more
> {code}
> Assigning to [~bharat] since he has already started work on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline
[
https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962313#comment-16962313
]
Bharat Viswanadham commented on HDDS-2356:
--
Hi [~timmylicheng]
{quote}The long printing logs happen only in your branch tho. The full log is
too huge to put here. You can think of the same logs as attached over and over
again for multi megabyte size.
{quote}
As said the log printing change is done in HDDS-2286 and as it is a list, we
add the same pipeline to it, it is printing that. Recently a change went in,
when the client got an error with pipeline, we exclude and used to fail if
there are no more pipelines, but now if there are no pipelines in the system
SCM ignores and reissues the same pipeline. Might be in your case, the same
pipeline is continuously excluded to some error, as this is the list we try to
add it. So this might be the reason for the huge big log statement. To avoid
this one thing which we can do is add to exclude list of pipelineId's if it
does not exist. Will open a Jira and fix this.
{quote}Also I'm using S3 gateway to connect to ozone and mount local file path
by fuse (goofys). Have you tested s3 gateway? Most unit tests are going thru
RPC.
{quote}
S3 Gateway also uses internally RpcClient to communicate to the cluster. If
possible, can we run the goofys in debug and get logs when completeMPU is
happening (what is the parameter list it passing and why this
MISMATCH_PART_LIST is happening) we can root cause this. I will also try to set
up a cluster and see it.
> Multipart upload report errors while writing to ozone Ratis pipeline
>
>
> Key: HDDS-2356
> URL: https://issues.apache.org/jira/browse/HDDS-2356
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Affects Versions: 0.4.1
> Environment: Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM
> on a separate VM
>Reporter: Li Cheng
>Assignee: Bharat Viswanadham
>Priority: Blocker
> Fix For: 0.5.0
>
>
> Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say
> it's VM0.
> I use goofys as a fuse and enable ozone S3 gateway to mount ozone to a path
> on VM0, while reading data from VM0 local disk and write to mount path. The
> dataset has various sizes of files from 0 byte to GB-level and it has a
> number of ~50,000 files.
> The writing is slow (1GB for ~10 mins) and it stops after around 4GB. As I
> look at hadoop-root-om-VM_50_210_centos.out log, I see OM throwing errors
> related with Multipart upload. This error eventually causes the writing to
> terminate and OM to be closed.
>
> 2019-10-24 16:01:59,527 [OMDoubleBufferFlushThread] ERROR - Terminating with
> exit status 2: OMDoubleBuffer flush
> threadOMDoubleBufferFlushThreadencountered Throwable error
> java.util.ConcurrentModificationException
> at java.util.TreeMap.forEach(TreeMap.java:1004)
> at
> org.apache.hadoop.ozone.om.helpers.OmMultipartKeyInfo.getProto(OmMultipartKeyInfo.java:111)
> at
> org.apache.hadoop.ozone.om.codec.OmMultipartKeyInfoCodec.toPersistedFormat(OmMultipartKeyInfoCodec.java:38)
> at
> org.apache.hadoop.ozone.om.codec.OmMultipartKeyInfoCodec.toPersistedFormat(OmMultipartKeyInfoCodec.java:31)
> at
> org.apache.hadoop.hdds.utils.db.CodecRegistry.asRawData(CodecRegistry.java:68)
> at
> org.apache.hadoop.hdds.utils.db.TypedTable.putWithBatch(TypedTable.java:125)
> at
> org.apache.hadoop.ozone.om.response.s3.multipart.S3MultipartUploadCommitPartResponse.addToDBBatch(S3MultipartUploadCommitPartResponse.java:112)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$flushTransactions$0(OzoneManagerDoubleBuffer.java:137)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:135)
> at java.lang.Thread.run(Thread.java:745)
> 2019-10-24 16:01:59,629 [shutdown-hook-0] INFO - SHUTDOWN_MSG:
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2379) OM terminates with RocksDB error while continuously writing keys.
Aravindan Vijayan created HDDS-2379:
---
Summary: OM terminates with RocksDB error while continuously
writing keys.
Key: HDDS-2379
URL: https://issues.apache.org/jira/browse/HDDS-2379
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: Ozone Manager
Reporter: Aravindan Vijayan
Assignee: Bharat Viswanadham
Fix For: 0.5.0
Exception trace after writing around 800,000 keys.
{code}
2019-10-29 11:15:15,131 ERROR
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer: Terminating with
exit status 1: During flush to DB encountered err
or in OMDoubleBuffer flush thread OMDoubleBufferFlushThread
java.io.IOException: Unable to write the batch.
at
org.apache.hadoop.hdds.utils.db.RDBBatchOperation.commit(RDBBatchOperation.java:48)
at
org.apache.hadoop.hdds.utils.db.RDBStore.commitBatchOperation(RDBStore.java:240)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:146)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.rocksdb.RocksDBException: unknown WriteBatch tag
at org.rocksdb.RocksDB.write0(Native Method)
at org.rocksdb.RocksDB.write(RocksDB.java:1421)
at
org.apache.hadoop.hdds.utils.db.RDBBatchOperation.commit(RDBBatchOperation.java:46)
... 3 more
{code}
Assigning to [~bharat] since he has already started work on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14927) RBF: Add metrics for async callers thread pool
[
https://issues.apache.org/jira/browse/HDFS-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962309#comment-16962309
]
Leon Gao commented on HDFS-14927:
-
[~inigoiri] Thanks for the review
Resolved 1) and 2) in your comments
{quote}Actually why do we do it asynchronously to then wait? Could we just do
the renewLease?
{quote}
I need to keep a renewlease call running in background to have an active thread
in the thread pool, so we can know the metrics work well.
{quote}Will this have timing issues if we rely on the executor to leave the
threads active for a short period? Maybe we should make the parameter that
leaves the thread active to be a long value
{quote}
I was following the similar timing pattern used in other tests in this class.
Some clients calling renewlease in the background and there is a downstream NN
with the slow response of 1~2 seconds to make the async call thread pool busy.
I think it is ok as if this one fails other previous tests should fail as well
:)
> RBF: Add metrics for async callers thread pool
> --
>
> Key: HDFS-14927
> URL: https://issues.apache.org/jira/browse/HDFS-14927
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Leon Gao
>Assignee: Leon Gao
>Priority: Minor
> Attachments: HDFS-14927.001.patch, HDFS-14927.002.patch,
> HDFS-14927.003.patch, HDFS-14927.004.patch, HDFS-14927.005.patch
>
>
> It is good to add some monitoring on the async caller thread pool to handle
> fan-out RPC client requests, so we know the utilization and when to bump up
> dfs.federation.router.client.thread-size
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2378) Remove "OZONE" as string used in tests
[
https://issues.apache.org/jira/browse/HDDS-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Chitlangia updated HDDS-2378:
Target Version/s: 0.5.0
> Remove "OZONE" as string used in tests
> --
>
> Key: HDDS-2378
> URL: https://issues.apache.org/jira/browse/HDDS-2378
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.4.1
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> Based on a review I have done a quick check, and there are quite a few places
> where we have hardcoded "ozone" as String literal or a capital version of it
> into the code.
> Let's check then one by one, and where it is possible replace it with
> OzoneConsts.OZONE, or if the lower case version is not acceptable at all
> places, then create an other constant with the uppercase version and use that.
> This is the search, and the results:
> {code:bash}
> find . -name *.java | while read FILE; do NUM=`grep -c -i "\"OZONE\"" $FILE`;
> if [ $NUM -gt 0 ]; then echo $FILE; fi; done | sort | uniq
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDataYaml.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestBlockManagerImpl.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java
> ./hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ServerUtils.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/SCMContainerManagerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/block/TestBlockManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestCloseContainerEventHandler.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSCMContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestContainerPlacement.java
> ./hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/CreateSubcommand.java
> ./hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/OzoneVersionInfo.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/metrics/TestSCMContainerManagerMetrics.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerOperations.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerStateMachineIdempotency.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestS3Shell.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestAllocateContainer.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSmallFile.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestGetCommittedBlockLengthAndPutKey.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientMetrics.java
> ./hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
>
[jira] [Updated] (HDDS-2378) Remove "OZONE" as string used in tests
[
https://issues.apache.org/jira/browse/HDDS-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Chitlangia updated HDDS-2378:
Affects Version/s: 0.4.1
> Remove "OZONE" as string used in tests
> --
>
> Key: HDDS-2378
> URL: https://issues.apache.org/jira/browse/HDDS-2378
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.4.1
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> Based on a review I have done a quick check, and there are quite a few places
> where we have hardcoded "ozone" as String literal or a capital version of it
> into the code.
> Let's check then one by one, and where it is possible replace it with
> OzoneConsts.OZONE, or if the lower case version is not acceptable at all
> places, then create an other constant with the uppercase version and use that.
> This is the search, and the results:
> {code:bash}
> find . -name *.java | while read FILE; do NUM=`grep -c -i "\"OZONE\"" $FILE`;
> if [ $NUM -gt 0 ]; then echo $FILE; fi; done | sort | uniq
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDataYaml.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestBlockManagerImpl.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java
> ./hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ServerUtils.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/SCMContainerManagerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/block/TestBlockManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestCloseContainerEventHandler.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSCMContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestContainerPlacement.java
> ./hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/CreateSubcommand.java
> ./hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/OzoneVersionInfo.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/metrics/TestSCMContainerManagerMetrics.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerOperations.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerStateMachineIdempotency.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestS3Shell.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestAllocateContainer.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSmallFile.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestGetCommittedBlockLengthAndPutKey.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientMetrics.java
> ./hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
>
[jira] [Updated] (HDDS-2378) Remove "OZONE" as string used in tests
[
https://issues.apache.org/jira/browse/HDDS-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Chitlangia updated HDDS-2378:
Component/s: test
> Remove "OZONE" as string used in tests
> --
>
> Key: HDDS-2378
> URL: https://issues.apache.org/jira/browse/HDDS-2378
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: test
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> Based on a review I have done a quick check, and there are quite a few places
> where we have hardcoded "ozone" as String literal or a capital version of it
> into the code.
> Let's check then one by one, and where it is possible replace it with
> OzoneConsts.OZONE, or if the lower case version is not acceptable at all
> places, then create an other constant with the uppercase version and use that.
> This is the search, and the results:
> {code:bash}
> find . -name *.java | while read FILE; do NUM=`grep -c -i "\"OZONE\"" $FILE`;
> if [ $NUM -gt 0 ]; then echo $FILE; fi; done | sort | uniq
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDataYaml.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestBlockManagerImpl.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java
> ./hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ServerUtils.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/SCMContainerManagerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/block/TestBlockManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestCloseContainerEventHandler.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSCMContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestContainerPlacement.java
> ./hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/CreateSubcommand.java
> ./hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/OzoneVersionInfo.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/metrics/TestSCMContainerManagerMetrics.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerOperations.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerStateMachineIdempotency.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestS3Shell.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestAllocateContainer.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSmallFile.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestGetCommittedBlockLengthAndPutKey.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientMetrics.java
> ./hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
>
[jira] [Commented] (HDDS-2378) Remove "OZONE" as string used in tests
[
https://issues.apache.org/jira/browse/HDDS-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962307#comment-16962307
]
Dinesh Chitlangia commented on HDDS-2378:
-
[~aengineer] This has stemmed from my review comments on
[https://github.com/apache/hadoop-ozone/pull/95]
[~pifta] went one step further to check it across the project instead and
offered to fix it.
I will definitely help to review/merge this.
> Remove "OZONE" as string used in tests
> --
>
> Key: HDDS-2378
> URL: https://issues.apache.org/jira/browse/HDDS-2378
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> Based on a review I have done a quick check, and there are quite a few places
> where we have hardcoded "ozone" as String literal or a capital version of it
> into the code.
> Let's check then one by one, and where it is possible replace it with
> OzoneConsts.OZONE, or if the lower case version is not acceptable at all
> places, then create an other constant with the uppercase version and use that.
> This is the search, and the results:
> {code:bash}
> find . -name *.java | while read FILE; do NUM=`grep -c -i "\"OZONE\"" $FILE`;
> if [ $NUM -gt 0 ]; then echo $FILE; fi; done | sort | uniq
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDataYaml.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestBlockManagerImpl.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java
> ./hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ServerUtils.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/SCMContainerManagerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/block/TestBlockManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestCloseContainerEventHandler.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSCMContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestContainerPlacement.java
> ./hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/CreateSubcommand.java
> ./hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/OzoneVersionInfo.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/metrics/TestSCMContainerManagerMetrics.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerOperations.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerStateMachineIdempotency.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestS3Shell.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestAllocateContainer.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSmallFile.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestGetCommittedBlockLengthAndPutKey.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientManager.java
>
[jira] [Commented] (HDDS-2378) Remove "OZONE" as string used in tests
[
https://issues.apache.org/jira/browse/HDDS-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962303#comment-16962303
]
Anu Engineer commented on HDDS-2378:
Not sure I understand the evil we are trying to prevent here. But if it
entertains you and you are willing to do this; please be my guest. I will
provide the review bandwidth :)
> Remove "OZONE" as string used in tests
> --
>
> Key: HDDS-2378
> URL: https://issues.apache.org/jira/browse/HDDS-2378
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> Based on a review I have done a quick check, and there are quite a few places
> where we have hardcoded "ozone" as String literal or a capital version of it
> into the code.
> Let's check then one by one, and where it is possible replace it with
> OzoneConsts.OZONE, or if the lower case version is not acceptable at all
> places, then create an other constant with the uppercase version and use that.
> This is the search, and the results:
> {code:bash}
> find . -name *.java | while read FILE; do NUM=`grep -c -i "\"OZONE\"" $FILE`;
> if [ $NUM -gt 0 ]; then echo $FILE; fi; done | sort | uniq
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDataYaml.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestBlockManagerImpl.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java
> ./hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ServerUtils.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/SCMContainerManagerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/block/TestBlockManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestCloseContainerEventHandler.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSCMContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestContainerPlacement.java
> ./hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/CreateSubcommand.java
> ./hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/OzoneVersionInfo.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/metrics/TestSCMContainerManagerMetrics.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerOperations.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerStateMachineIdempotency.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestS3Shell.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestAllocateContainer.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSmallFile.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestGetCommittedBlockLengthAndPutKey.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientMetrics.java
>
[jira] [Commented] (HDFS-14284) RBF: Log Router identifier when reporting exceptions
[
https://issues.apache.org/jira/browse/HDFS-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962301#comment-16962301
]
Hadoop QA commented on HDFS-14284:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
36s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14284 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12984292/HDFS-14284.008.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ddf4517e923b 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bd498ba |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28204/testReport/ |
| Max. process+thread count | 2760 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28204/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Log Router identifier when reporting exceptions
>
>
> Key: HDFS-14284
> URL:
[jira] [Commented] (HDFS-14936) Add getNumOfChildren() for interface InnerNode
[ https://issues.apache.org/jira/browse/HDFS-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962294#comment-16962294 ] Íñigo Goiri commented on HDFS-14936: +1 on [^HDFS-14936.003.patch]. > Add getNumOfChildren() for interface InnerNode > -- > > Key: HDFS-14936 > URL: https://issues.apache.org/jira/browse/HDFS-14936 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Minor > Attachments: HDFS-14936.001.patch, HDFS-14936.002.patch, > HDFS-14936.003.patch > > > current code InnerNode subclass InnerNodeImpl and DFSTopologyNodeImpl both > have getNumOfChildren(). > so Add getNumOfChildren() for interface InnerNode and remove unnessary > getNumOfChildren() in DFSTopologyNodeImpl. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2364) Add a OM metrics to find the false positive rate for the keyMayExist
[ https://issues.apache.org/jira/browse/HDDS-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-2364: --- Assignee: Aravindan Vijayan > Add a OM metrics to find the false positive rate for the keyMayExist > > > Key: HDDS-2364 > URL: https://issues.apache.org/jira/browse/HDDS-2364 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.5.0 >Reporter: Mukul Kumar Singh >Assignee: Aravindan Vijayan >Priority: Major > > Add a OM metrics to find the false positive rate for the keyMayExist. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13507) RBF: Remove update functionality from routeradmin's add cmd
[ https://issues.apache.org/jira/browse/HDFS-13507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962288#comment-16962288 ] Ayush Saxena commented on HDFS-13507: - Along with removing update functionality, we should at the same time ensure as I said before too, we should allow different targets. Which we were able to do before by using add command multiple times. > RBF: Remove update functionality from routeradmin's add cmd > --- > > Key: HDFS-13507 > URL: https://issues.apache.org/jira/browse/HDFS-13507 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei Yan >Assignee: Gang Li >Priority: Minor > Labels: incompatible > Attachments: HDFS-13507-HDFS-13891.003.patch, > HDFS-13507-HDFS-13891.004.patch, HDFS-13507.000.patch, HDFS-13507.001.patch, > HDFS-13507.002.patch, HDFS-13507.003.patch > > > Follow up the discussion in HDFS-13326. We should remove the "update" > functionality from routeradmin's add cmd, to make it consistent with RPC > calls. > Note that: this is an incompatible change. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13507) RBF: Remove update functionality from routeradmin's add cmd
[ https://issues.apache.org/jira/browse/HDFS-13507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962283#comment-16962283 ] hemanthboyina commented on HDFS-13507: -- submitted [^HDFS-13507.003.patch] > RBF: Remove update functionality from routeradmin's add cmd > --- > > Key: HDFS-13507 > URL: https://issues.apache.org/jira/browse/HDFS-13507 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei Yan >Assignee: Gang Li >Priority: Minor > Labels: incompatible > Attachments: HDFS-13507-HDFS-13891.003.patch, > HDFS-13507-HDFS-13891.004.patch, HDFS-13507.000.patch, HDFS-13507.001.patch, > HDFS-13507.002.patch, HDFS-13507.003.patch > > > Follow up the discussion in HDFS-13326. We should remove the "update" > functionality from routeradmin's add cmd, to make it consistent with RPC > calls. > Note that: this is an incompatible change. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2378) Remove "OZONE" as string used in tests
[
https://issues.apache.org/jira/browse/HDDS-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Istvan Fajth reassigned HDDS-2378:
--
Assignee: Istvan Fajth
> Remove "OZONE" as string used in tests
> --
>
> Key: HDDS-2378
> URL: https://issues.apache.org/jira/browse/HDDS-2378
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> Based on a review I have done a quick check, and there are quite a few places
> where we have hardcoded "ozone" as String literal or a capital version of it
> into the code.
> Let's check then one by one, and where it is possible replace it with
> OzoneConsts.OZONE, or if the lower case version is not acceptable at all
> places, then create an other constant with the uppercase version and use that.
> This is the search, and the results:
> {code:bash}
> find . -name *.java | while read FILE; do NUM=`grep -c -i "\"OZONE\"" $FILE`;
> if [ $NUM -gt 0 ]; then echo $FILE; fi; done | sort | uniq
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDataYaml.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestBlockManagerImpl.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java
> ./hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ServerUtils.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/SCMContainerManagerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/block/TestBlockManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestCloseContainerEventHandler.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSCMContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestContainerPlacement.java
> ./hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/CreateSubcommand.java
> ./hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/OzoneVersionInfo.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/metrics/TestSCMContainerManagerMetrics.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerOperations.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerStateMachineIdempotency.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestS3Shell.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestAllocateContainer.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSmallFile.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestGetCommittedBlockLengthAndPutKey.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientMetrics.java
> ./hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
> ./hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestS3BucketManager.java
>
[jira] [Updated] (HDFS-13507) RBF: Remove update functionality from routeradmin's add cmd
[ https://issues.apache.org/jira/browse/HDFS-13507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina updated HDFS-13507: - Attachment: HDFS-13507.003.patch > RBF: Remove update functionality from routeradmin's add cmd > --- > > Key: HDFS-13507 > URL: https://issues.apache.org/jira/browse/HDFS-13507 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei Yan >Assignee: Gang Li >Priority: Minor > Labels: incompatible > Attachments: HDFS-13507-HDFS-13891.003.patch, > HDFS-13507-HDFS-13891.004.patch, HDFS-13507.000.patch, HDFS-13507.001.patch, > HDFS-13507.002.patch, HDFS-13507.003.patch > > > Follow up the discussion in HDFS-13326. We should remove the "update" > functionality from routeradmin's add cmd, to make it consistent with RPC > calls. > Note that: this is an incompatible change. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2378) Remove "OZONE" as string used in tests
Istvan Fajth created HDDS-2378:
--
Summary: Remove "OZONE" as string used in tests
Key: HDDS-2378
URL: https://issues.apache.org/jira/browse/HDDS-2378
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Reporter: Istvan Fajth
Based on a review I have done a quick check, and there are quite a few places
where we have hardcoded "ozone" as String literal or a capital version of it
into the code.
Let's check then one by one, and where it is possible replace it with
OzoneConsts.OZONE, or if the lower case version is not acceptable at all
places, then create an other constant with the uppercase version and use that.
This is the search, and the results:
{code:bash}
find . -name *.java | while read FILE; do NUM=`grep -c -i "\"OZONE\"" $FILE`;
if [ $NUM -gt 0 ]; then echo $FILE; fi; done | sort | uniq
./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java
./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
./hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDataYaml.java
./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestBlockManagerImpl.java
./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java
./hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ServerUtils.java
./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/SCMContainerManagerMetrics.java
./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementMetrics.java
./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeMetrics.java
./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineMetrics.java
./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java
./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/block/TestBlockManager.java
./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestCloseContainerEventHandler.java
./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSCMContainerManager.java
./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestContainerPlacement.java
./hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/CreateSubcommand.java
./hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/OzoneVersionInfo.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/metrics/TestSCMContainerManagerMetrics.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerOperations.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerStateMachineIdempotency.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManager.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestS3Shell.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestAllocateContainer.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSmallFile.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestGetCommittedBlockLengthAndPutKey.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientManager.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientMetrics.java
./hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
./hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestS3BucketManager.java
./hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBufferWithOMResponse.java
./hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMAllocateBlockRequest.java
./hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCommitRequest.java
[jira] [Updated] (HDFS-14284) RBF: Log Router identifier when reporting exceptions
[ https://issues.apache.org/jira/browse/HDFS-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina updated HDFS-14284: - Attachment: HDFS-14284.008.patch > RBF: Log Router identifier when reporting exceptions > > > Key: HDFS-14284 > URL: https://issues.apache.org/jira/browse/HDFS-14284 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-14284.001.patch, HDFS-14284.002.patch, > HDFS-14284.003.patch, HDFS-14284.004.patch, HDFS-14284.005.patch, > HDFS-14284.006.patch, HDFS-14284.007.patch, HDFS-14284.008.patch > > > The typical setup is to use multiple Routers through > ConfiguredFailoverProxyProvider. > In a regular HA Namenode setup, it is easy to know which NN was used. > However, in RBF, any Router can be the one reporting the exception and it is > hard to know which was the one. > We should have a way to identify which Router/Namenode was the one triggering > the exception. > This would also apply with Observer Namenodes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14920) Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission
[
https://issues.apache.org/jira/browse/HDFS-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962217#comment-16962217
]
Hadoop QA commented on HDFS-14920:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 147 unchanged - 0 fixed = 148 total (was 147) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 55s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSClientRetries |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14920 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12984274/HDFS-14920.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 3ca37fdf6f18 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ed45c13 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/28203/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/28203/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28203/testReport/ |
| Max. process+thread count | 3362 (vs. ulimit of 5500) |
| modules | C:
[jira] [Commented] (HDDS-2372) Datanode pipeline is failing with NoSuchFileException
[
https://issues.apache.org/jira/browse/HDDS-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962159#comment-16962159
]
Marton Elek commented on HDDS-2372:
---
I tried to reproduce it locally with docker-compose. In container state machine
I reduced the capacity of the cache:
{code:java}
stateMachineDataCache = CacheBuilder.newBuilder()
.expireAfterAccess(500, TimeUnit.MILLISECONDS)
// set the limit on no of cached entries equal to no of max threads
// executing writeStateMachineData
.maximumSize(10).build();
{code}
And added a random wait to the readStateMachine:
{code:java}
private ByteString readStateMachineData(
ContainerCommandRequestProto requestProto, long term, long index)
throws IOException {
if (Math.random() > 0.7) {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
} {code}
I got similar, but different error:
{code:java}
-SegmentedRaftLogWorker: created new log segment
/data/metadata/ratis/68c226d2-356c-4eb0-aee2-ce458d4b0095/current/log_inprogress_6872
ESC[32mdatanode_3|ESC[0m 2019-10-29 15:54:10,084 [pool-7-thread-38] ERROR
- Unable to find the chunk file. chunk info :
ChunkInfo{chunkName='3D4nM8ycqh_testdata_chunk_4366, offset=0, len=1024}
ESC[32mdatanode_3|ESC[0m 2019-10-29 15:54:10,085 [pool-7-thread-38] INFO
- Operation: ReadChunk : Trace ID:
b93bcdcdd7fd37c:a3bed642046e9e09:b93bcdcdd7fd37c:1 : Message: Unable to find
the chunk file. chunk info ChunkInfo{chunkName='3D4nM8ycqh_testdata_chunk_4366,
offset=0, len=1024} : Result: UNABLE_TO_FIND_CHUNK
ESC[32mdatanode_3|ESC[0m 2019-10-29 15:54:10,085 [pool-7-thread-38] ERROR
- gid group-CE458D4B0095 : ReadStateMachine failed. cmd ReadChunk logIndex
8773 msg : Unable to find the chunk file. chunk info
ChunkInfo{chunkName='3D4nM8ycqh_testdata_chunk_4366, offset=0, len=1024}
Container Result: UNABLE_TO_FIND_CHUNK
ESC[32mdatanode_3|ESC[0m 2019-10-29 15:54:10,086 ERROR raftlog.RaftLog:
06f4231d-30a8-42fd-839e-aeaea7b1aa72@group-CE458D4B0095-SegmentedRaftLog:
Failed readStateMachineData for (t:2, i:8773), STATEMACHINELOGENTRY,
client-BCA58E609475, cid=4367
ESC[32mdatanode_3|ESC[0m java.util.concurrent.ExecutionException:
java.util.concurrent.ExecutionException:
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
Unable to find the chunk file. chunk info
ChunkInfo{chunkName='3D4nM8ycqh_testdata_chunk_4366, offset=0, len=1024}
ESC[32mdatanode_3|ESC[0mat
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
ESC[32mdatanode_3|ESC[0mat
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022)
ESC[32mdatanode_3|ESC[0mat
org.apache.ratis.server.raftlog.RaftLog$EntryWithData.getEntry(RaftLog.java:472)
ESC[32mdatanode_3|ESC[0mat
org.apache.ratis.util.DataQueue.pollList(DataQueue.java:134)
ESC[32mdatanode_3|ESC[0mat
org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:220)
ESC[32mdatanode_3|ESC[0mat
org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:178)
ESC[32mdatanode_3|ESC[0mat
org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:121)
ESC[32mdatanode_3|ESC[0mat
org.apache.ratis.server.impl.LogAppender$AppenderDaemon.run(LogAppender.java:76)
ESC[32mdatanode_3|ESC[0mat
java.base/java.lang.Thread.run(Thread.java:834) {code}
And the cluster is stuck in a bad state (couln't write any more chunk, ever)
{code:java}
datanode_1| 2019-10-29 15:54:10,099 INFO impl.RaftServerImpl:
6b9ca1af-467f-40c7-a21d-118cb34080b1@group-CE458D4B0095: inconsistency entries.
Reply:06f4231d-30a8-42fd-839e-aeaea7b1aa72<-6b9ca1af-467f-40c7-a21d-118cb34080b1#0:FAIL,INCONSISTENCY,nextIndex:8773,term:2,followerCommit:8768
{code}
Fix me If I am wrong, but
* I think the write path should work even if the cache is limited or there are
unexpected sleep
* If there are some inconsistencies the raft ring should be healed or closed
and reopened (but it's an independent issue)
> Datanode pipeline is failing with NoSuchFileException
> -
>
> Key: HDDS-2372
> URL: https://issues.apache.org/jira/browse/HDDS-2372
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Marton Elek
>Priority: Critical
>
> Found it on a k8s based test cluster using a simple 3 node cluster and
> HDDS-2327 freon test. After a while the StateMachine become unhealthy after
> this error:
> {code:java}
> datanode-0 datanode java.util.concurrent.ExecutionException:
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> java.nio.file.NoSuchFileException:
>
[jira] [Commented] (HDDS-2372) Datanode pipeline is failing with NoSuchFileException
[
https://issues.apache.org/jira/browse/HDDS-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962153#comment-16962153
]
Marton Elek commented on HDDS-2372:
---
Thanks the help [~szetszwo]
# I found it only at one datanode. But it's hard to reproduce, usually I need
to write a lot of write chunks
# Yes, the test writes chunks to one ratis pipeline without using any real
block id / container id. It's uploaded in HDDS-2327 (Use patch + ozone freon
dcg -n 10)
# Yes, this is the logic in ChunkManagerImpl.readChunk but I can't see any
lock / sync between checking the files. Chunk can be committed in the middle of
the read / tests (IMHO)
{code:java}
if (containerData.getLayOutVersion() == ChunkLayOutVersion
.getLatestVersion().getVersion()) {
File chunkFile = ChunkUtils.getChunkFile(containerData, info);
// In case the chunk file does not exist but tmp chunk file exist,
// read from tmp chunk file if readFromTmpFile is set to true
if (!chunkFile.exists() && dispatcherContext != null
&& dispatcherContext.isReadFromTmpFile()) {
//WHAT IF CHUNK IS COMMITTED AT THIS POINT?
chunkFile = getTmpChunkFile(chunkFile, dispatcherContext);
}
data = ChunkUtils.readData(chunkFile, info, volumeIOStats); {code}
> Datanode pipeline is failing with NoSuchFileException
> -
>
> Key: HDDS-2372
> URL: https://issues.apache.org/jira/browse/HDDS-2372
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Marton Elek
>Priority: Critical
>
> Found it on a k8s based test cluster using a simple 3 node cluster and
> HDDS-2327 freon test. After a while the StateMachine become unhealthy after
> this error:
> {code:java}
> datanode-0 datanode java.util.concurrent.ExecutionException:
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> java.nio.file.NoSuchFileException:
> /data/storage/hdds/2a77fab9-9dc5-4f73-9501-b5347ac6145c/current/containerDir0/1/chunks/gGYYgiTTeg_testdata_chunk_13931.tmp.2.20830
> {code}
> Can be reproduced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14939) Update documnet and UT tests for Quota API.
[
https://issues.apache.org/jira/browse/HDFS-14939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hemanthboyina reassigned HDFS-14939:
Assignee: hemanthboyina
> Update documnet and UT tests for Quota API.
> ---
>
> Key: HDFS-14939
> URL: https://issues.apache.org/jira/browse/HDFS-14939
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Surendra Singh Lilhore
>Assignee: hemanthboyina
>Priority: Major
>
> Refer comment in HDFS-8631.
> {quote} # mention to me when you're going near this class as I can make
> suggestions in advance.
> # I now expect the documentation and the extra testing. Who is going to
> volunteer to do this?{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14940) HDFS Balancer : getBalancerBandwidth displaying wrong values for the maximum network bandwidth used by the datanode while network bandwidth set with values as 104857600
[ https://issues.apache.org/jira/browse/HDFS-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962133#comment-16962133 ] hemanthboyina commented on HDFS-14940: -- Thanks for putting this up [~SouryakantaDwivedy] , will work on this > HDFS Balancer : getBalancerBandwidth displaying wrong values for the maximum > network bandwidth used by the datanode while network bandwidth set with > values as 1048576000g/1048p/1e > --- > > Key: HDFS-14940 > URL: https://issues.apache.org/jira/browse/HDFS-14940 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer mover >Affects Versions: 3.1.1 > Environment: 3 Node HA Setup >Reporter: Souryakanta Dwivedy >Priority: Minor > Attachments: BalancerBW.PNG > > > HDFS Balancer : getBalancerBandwidth displaying wrong values for the maximum > network bandwidth used by the datanode > while network bandwidth set with values as 1048576000g/1048p/1e > Steps :- > * Set balancer bandwith with command setBalancerBandwidth and vlaues as > [1048576000g/1048p/1e] > * - Check bandwidth used by the datanode during HDFS block balancing with > command :hdfs dfsadmin -getBalancerBandwidth " check it will display some > different values not the same value as set -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14907) [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
[
https://issues.apache.org/jira/browse/HDFS-14907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962125#comment-16962125
]
Hudson commented on HDFS-14907:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17582 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17582/])
HDFS-14907. [Dynamometer] Add JUnit JAR to classpath for (xkrogen: rev
e32ab5e179bd32f8c18107536c15e577cf93d435)
* (edit)
hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/resources/start-component.sh
> [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
> --
>
> Key: HDFS-14907
> URL: https://issues.apache.org/jira/browse/HDFS-14907
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.3.0
>
>
> When executing {{start-dynamometer-cluster.sh}} with Hadoop-3 binary,
> datanodes fail to run with the following log and
> {{start-dynamometer-cluster.sh}} fails.
> {noformat}
> LogType:stderr
> LogLastModifiedTime:Wed Oct 09 15:03:09 +0900 2019
> LogLength:1386
> LogContents:
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:299)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:243)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:252)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2982)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2972)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.formatDataNodeDirs(MiniDFSCluster.java:2834)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(SimulatedDataNodes.java:123)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.main(SimulatedDataNodes.java:88)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 9 more
> ./start-component.sh: line 317: kill: (2261) - No such process
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14824) [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the benchmark results.
[ https://issues.apache.org/jira/browse/HDFS-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962124#comment-16962124 ] Erik Krogen commented on HDFS-14824: Hi [~tasanuma], if you took this up it would be much appreciated! :) > [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the > benchmark results. > --- > > Key: HDFS-14824 > URL: https://issues.apache.org/jira/browse/HDFS-14824 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Soya Miyoshi >Priority: Major > > According to the latest > [document|https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-dynamometer/Dynamometer.html > ], the benchmark results will be written in `Dauditreplay.output-path`. > However, current org.apache.hadooop.tools hasn't merged [this pull > request|https://github.com/linkedin/dynamometer/pull/76 ], so it does not > output the benchmark results. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14907) [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
[
https://issues.apache.org/jira/browse/HDFS-14907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962122#comment-16962122
]
Erik Krogen commented on HDFS-14907:
I just committed this to trunk. Thanks for the contribution [~tasanuma]!
> [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
> --
>
> Key: HDFS-14907
> URL: https://issues.apache.org/jira/browse/HDFS-14907
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
>
> When executing {{start-dynamometer-cluster.sh}} with Hadoop-3 binary,
> datanodes fail to run with the following log and
> {{start-dynamometer-cluster.sh}} fails.
> {noformat}
> LogType:stderr
> LogLastModifiedTime:Wed Oct 09 15:03:09 +0900 2019
> LogLength:1386
> LogContents:
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:299)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:243)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:252)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2982)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2972)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.formatDataNodeDirs(MiniDFSCluster.java:2834)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(SimulatedDataNodes.java:123)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.main(SimulatedDataNodes.java:88)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 9 more
> ./start-component.sh: line 317: kill: (2261) - No such process
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14907) [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
[
https://issues.apache.org/jira/browse/HDFS-14907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-14907:
---
Fix Version/s: 3.3.0
> [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
> --
>
> Key: HDFS-14907
> URL: https://issues.apache.org/jira/browse/HDFS-14907
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.3.0
>
>
> When executing {{start-dynamometer-cluster.sh}} with Hadoop-3 binary,
> datanodes fail to run with the following log and
> {{start-dynamometer-cluster.sh}} fails.
> {noformat}
> LogType:stderr
> LogLastModifiedTime:Wed Oct 09 15:03:09 +0900 2019
> LogLength:1386
> LogContents:
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:299)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:243)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:252)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2982)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2972)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.formatDataNodeDirs(MiniDFSCluster.java:2834)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(SimulatedDataNodes.java:123)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.main(SimulatedDataNodes.java:88)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 9 more
> ./start-component.sh: line 317: kill: (2261) - No such process
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14936) Add getNumOfChildren() for interface InnerNode
[
https://issues.apache.org/jira/browse/HDFS-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962117#comment-16962117
]
Hadoop QA commented on HDFS-14936:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
5s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
23m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 21m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
14s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}119m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}270m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSInputStream |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14936 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12984246/HDFS-14936.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux bae82c633de4 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ed45c13 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
[jira] [Commented] (HDDS-2372) Datanode pipeline is failing with NoSuchFileException
[
https://issues.apache.org/jira/browse/HDDS-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962112#comment-16962112
]
Tsz-wo Sze commented on HDDS-2372:
--
Some questions (sorry that I don't understand the test):
- Did the NoSuchFileException happen in all three data nodes? Or just one?
- What did the test do? Writing a lot of chunks to one Ratis pipeline?
- Did the read in B.3 fail? It sounds like yes according to "the chunk can't
be read any more from the tmp file." Was the tmp file moved to another
location? If yes, the read should also try reading from there.
Since this can be reproduced, we should add more log messages to trace back
when did the tmp file get created, moved/deleted.
> Datanode pipeline is failing with NoSuchFileException
> -
>
> Key: HDDS-2372
> URL: https://issues.apache.org/jira/browse/HDDS-2372
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Marton Elek
>Priority: Critical
>
> Found it on a k8s based test cluster using a simple 3 node cluster and
> HDDS-2327 freon test. After a while the StateMachine become unhealthy after
> this error:
> {code:java}
> datanode-0 datanode java.util.concurrent.ExecutionException:
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> java.nio.file.NoSuchFileException:
> /data/storage/hdds/2a77fab9-9dc5-4f73-9501-b5347ac6145c/current/containerDir0/1/chunks/gGYYgiTTeg_testdata_chunk_13931.tmp.2.20830
> {code}
> Can be reproduced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2232) ozone-build docker image is failing due to a missing entrypoint script
[ https://issues.apache.org/jira/browse/HDDS-2232?focusedWorklogId=335554=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-335554 ] ASF GitHub Bot logged work on HDDS-2232: Author: ASF GitHub Bot Created on: 29/Oct/19 15:07 Start Date: 29/Oct/19 15:07 Worklog Time Spent: 10m Work Description: elek commented on pull request #6: HDDS-2232. ozone-build docker image is failing due to a missing entry… URL: https://github.com/apache/hadoop-docker-ozone/pull/6 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 335554) Time Spent: 20m (was: 10m) > ozone-build docker image is failing due to a missing entrypoint script > -- > > Key: HDDS-2232 > URL: https://issues.apache.org/jira/browse/HDDS-2232 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: build, docker >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > We have a dedicated apache/ozone-build image which contains all of the > required build and test tools to build ozone. > Unfortunately it's not working as one shell script was not added to the > original patch. > This patch (to the hadoop-docker-ozone repo!!) remove the requirement of the > entrypoint.sh (no more docker in docker) > > And installs additional tools (blockade, kubectl, mailsend) > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2232) ozone-build docker image is failing due to a missing entrypoint script
[ https://issues.apache.org/jira/browse/HDDS-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek updated HDDS-2232: -- Resolution: Fixed Status: Resolved (was: Patch Available) > ozone-build docker image is failing due to a missing entrypoint script > -- > > Key: HDDS-2232 > URL: https://issues.apache.org/jira/browse/HDDS-2232 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: build, docker >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > We have a dedicated apache/ozone-build image which contains all of the > required build and test tools to build ozone. > Unfortunately it's not working as one shell script was not added to the > original patch. > This patch (to the hadoop-docker-ozone repo!!) remove the requirement of the > entrypoint.sh (no more docker in docker) > > And installs additional tools (blockade, kubectl, mailsend) > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14936) Add getNumOfChildren() for interface InnerNode
[ https://issues.apache.org/jira/browse/HDFS-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962090#comment-16962090 ] Siyao Meng commented on HDFS-14936: --- Thanks [~leosun08]. +1 on rev 003 > Add getNumOfChildren() for interface InnerNode > -- > > Key: HDFS-14936 > URL: https://issues.apache.org/jira/browse/HDFS-14936 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Minor > Attachments: HDFS-14936.001.patch, HDFS-14936.002.patch, > HDFS-14936.003.patch > > > current code InnerNode subclass InnerNodeImpl and DFSTopologyNodeImpl both > have getNumOfChildren(). > so Add getNumOfChildren() for interface InnerNode and remove unnessary > getNumOfChildren() in DFSTopologyNodeImpl. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14938) Add check if excludedNodes contain scope in DFSNetworkTopology#chooseRandomWithStorageType()
[
https://issues.apache.org/jira/browse/HDFS-14938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962076#comment-16962076
]
Hadoop QA commented on HDFS-14938:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 49s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 30s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}151m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.client.impl.TestBlockReaderLocal |
| | hadoop.hdfs.client.impl.TestBlockReaderFactory |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14938 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12984255/HDFS-14938.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 4f6ce66fbe2b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ed45c13 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/28202/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28202/testReport/ |
| Max. process+thread count | 3738 (vs. ulimit of 5500) |
| modules | C:
