[jira] [Commented] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[
https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842897#comment-16842897
]
Hadoop QA commented on HDFS-14431:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
3s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 17s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch
generated 4 new + 7 unchanged - 0 fixed = 11 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 24m 13s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRouterFaultTolerant |
| | hadoop.hdfs.server.federation.router.TestRouterAdminCLI |
| |
hadoop.hdfs.server.federation.router.TestRouterRPCMultipleDestinationMountTableResolver
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14431 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12969060/HDFS-14431-HDFS-13891.007.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux bcd11fd4adca 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / 09f39bf |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26803/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| unit |
[jira] [Commented] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[
https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842846#comment-16842846
]
Hadoop QA commented on HDFS-14353:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 15s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 2s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestReconstructStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14353 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12969050/HDFS-14353.009.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8fd2976e14d9 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 12c8161 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26802/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26802/testReport/ |
| Max. process+thread count | 2799 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Commented] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[
https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842836#comment-16842836
]
Íñigo Goiri commented on HDFS-14431:
The tests are now complete.
There are a couple issues that are solved but open to feedback:
* HASH needs to have the folders in all subclusters if we need to rename a
file. Changed {{DestinationOrder}}.
* When renaming with OVERWRITE, we may need to delete the file from another
subcluster. This is done in {{RenameParam#getLocationsToDelete()}}.
> RBF: Rename with multiple subclusters should fail if no eligible locations
> --
>
> Key: HDFS-14431
> URL: https://issues.apache.org/jira/browse/HDFS-14431
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-14431-HDFS-13891.001.patch,
> HDFS-14431-HDFS-13891.002.patch, HDFS-14431-HDFS-13891.003.patch,
> HDFS-14431-HDFS-13891.004.patch, HDFS-14431-HDFS-13891.005.patch,
> HDFS-14431-HDFS-13891.006.patch, HDFS-14431-HDFS-13891.007.patch
>
>
> Currently, the rename will fail with FileNotFoundException which is not clear
> to the user.
> The operation should fail stating the reason is that there are no eligible
> destinations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[ https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14431: --- Attachment: HDFS-14431-HDFS-13891.007.patch > RBF: Rename with multiple subclusters should fail if no eligible locations > -- > > Key: HDFS-14431 > URL: https://issues.apache.org/jira/browse/HDFS-14431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14431-HDFS-13891.001.patch, > HDFS-14431-HDFS-13891.002.patch, HDFS-14431-HDFS-13891.003.patch, > HDFS-14431-HDFS-13891.004.patch, HDFS-14431-HDFS-13891.005.patch, > HDFS-14431-HDFS-13891.006.patch, HDFS-14431-HDFS-13891.007.patch > > > Currently, the rename will fail with FileNotFoundException which is not clear > to the user. > The operation should fail stating the reason is that there are no eligible > destinations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1551) Implement Bucket Write Requests to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1551?focusedWorklogId=244460=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244460 ] ASF GitHub Bot logged work on HDDS-1551: Author: ASF GitHub Bot Created on: 18/May/19 01:03 Start Date: 18/May/19 01:03 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #827: HDDS-1551. Implement Bucket Write Requests to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/827#discussion_r285322909 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java ## @@ -313,6 +313,7 @@ private OzoneManager(OzoneConfiguration conf) throws IOException, RPC.setProtocolEngine(configuration, OzoneManagerProtocolPB.class, ProtobufRpcEngine.class); +metadataManager = new OmMetadataManagerImpl(configuration); Review comment: whitespace:end of line This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244460) Time Spent: 1.5h (was: 1h 20m) > Implement Bucket Write Requests to use Cache and DoubleBuffer > - > > Key: HDDS-1551 > URL: https://issues.apache.org/jira/browse/HDDS-1551 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Implement Bucket write requests to use OM Cache, double buffer. > And also in OM previously we used to Ratis client for communication to Ratis > server, instead of that use Ratis server API's. > > In this Jira will add the changes to implement bucket operations, and > HA/Non-HA will have a different code path, but once all requests are > implemented will have a single code path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1551) Implement Bucket Write Requests to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1551?focusedWorklogId=244461=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244461 ] ASF GitHub Bot logged work on HDDS-1551: Author: ASF GitHub Bot Created on: 18/May/19 01:03 Start Date: 18/May/19 01:03 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #827: HDDS-1551. Implement Bucket Write Requests to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/827#issuecomment-493636338 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 25 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 13 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 72 | Maven dependency ordering for branch | | +1 | mvninstall | 415 | trunk passed | | +1 | compile | 195 | trunk passed | | +1 | checkstyle | 52 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 878 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 121 | trunk passed | | 0 | spotbugs | 251 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 434 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 59 | Maven dependency ordering for patch | | +1 | mvninstall | 392 | the patch passed | | +1 | compile | 205 | the patch passed | | +1 | cc | 205 | the patch passed | | +1 | javac | 205 | the patch passed | | +1 | checkstyle | 56 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | -1 | whitespace | 0 | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 | shadedclient | 711 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 62 | hadoop-ozone generated 4 new + 2 unchanged - 0 fixed = 6 total (was 2) | | +1 | findbugs | 440 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 148 | hadoop-hdds in the patch failed. | | -1 | unit | 1101 | hadoop-ozone in the patch failed. | | +1 | asflicense | 33 | The patch does not generate ASF License warnings. | | | | 5546 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-827/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/827 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 68eb0f62efa3 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct 31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 12c8161 | | Default Java | 1.8.0_212 | | whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-827/4/artifact/out/whitespace-eol.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-827/4/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-827/4/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-827/4/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-827/4/testReport/ | | Max. process+thread count | 3972 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-827/4/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244461) Time Spent: 1h 40m (was: 1.5h) > Implement Bucket Write Requests to use Cache and DoubleBuffer > - > > Key: HDDS-1551 > URL: https://issues.apache.org/jira/browse/HDDS-1551 >
[jira] [Commented] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842825#comment-16842825
]
Hadoop QA commented on HDDS-1458:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 7m
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
1s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:blue}0{color} | {color:blue} yamllint {color} | {color:blue} 0m
0s{color} | {color:blue} yamllint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 19m
29s{color} | {color:green} trunk passed {color} |
| {color:orange}-0{color} | {color:orange} pylint {color} | {color:orange} 0m
4s{color} | {color:orange} Error running pylint. Please check pylint stderr
files. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} hadolint {color} | {color:red} 0m
3s{color} | {color:red} The patch generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 17m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} pylint {color} | {color:orange} 0m
8s{color} | {color:orange} Error running pylint. Please check pylint stderr
files. {color} |
| {color:green}+1{color} | {color:green} pylint {color} | {color:green} 0m
9s{color} | {color:green} There were no new pylint issues. {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
1s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
16s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 33s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}195m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/PreCommit-HDDS-Build/2698/artifact/out/Dockerfile
|
| JIRA Issue | HDDS-1458
[jira] [Commented] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842817#comment-16842817
]
Eric Yang commented on HDDS-1458:
-
[~arpitagarwal] Some of the commands run by freon is expecting an input, but I
see there is no pseudo-terminal setup for freon to call out to ozone commands.
Some code in freon needs to update to setup
[pseudo-terminal|https://stackoverflow.com/questions/41542960/run-interactive-bash-with-popen-and-a-dedicated-tty-python].
Can we have that improvement track by another issue?
Patch 8 made the following changes:
# Fixed start-build-env.sh change to work properly on MacOSX, (Found by [~jnp])
# Removed disk tests. (requested by [~elek]
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Attachments: HDDS-1458.001.patch, HDDS-1458.002.patch,
> HDDS-1458.003.patch, HDDS-1458.004.patch, HDDS-1458.005.patch,
> HDDS-1458.006.patch, HDDS-1458.007.patch, HDDS-1458.008.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12766) Ozone: Ensure usage of parameterized slf4j log syntax for ozone
[
https://issues.apache.org/jira/browse/HDFS-12766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham reassigned HDFS-12766:
-
Assignee: (was: Bharat Viswanadham)
> Ozone: Ensure usage of parameterized slf4j log syntax for ozone
> ---
>
> Key: HDFS-12766
> URL: https://issues.apache.org/jira/browse/HDFS-12766
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Xiaoyu Yao
>Priority: Trivial
> Labels: newbie
> Fix For: HDFS-7240
>
>
> various places use LOG.info("text " + something). they should all move to
> LOG.info("text {}", something)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12765) Ozone: Add proper timeout for all ozone tests
[ https://issues.apache.org/jira/browse/HDFS-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham reassigned HDFS-12765: - Assignee: (was: Bharat Viswanadham) > Ozone: Add proper timeout for all ozone tests > - > > Key: HDFS-12765 > URL: https://issues.apache.org/jira/browse/HDFS-12765 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Xiaoyu Yao >Priority: Major > Fix For: HDFS-7240 > > > Proper @Timeout annotation to guarantee ozone tests will not hold the Jenkins > machines too long. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842801#comment-16842801 ] maobaolong commented on HDFS-14353: --- [~elgoiri] Thanking for remind me, i have corrected it. PTAL. > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14353.001.patch, HDFS-14353.002.patch, > HDFS-14353.003.patch, HDFS-14353.004.patch, HDFS-14353.005.patch, > HDFS-14353.006.patch, HDFS-14353.007.patch, HDFS-14353.008.patch, > HDFS-14353.009.patch, screenshot-1.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[
https://issues.apache.org/jira/browse/HDDS-1530?focusedWorklogId=244426=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244426
]
ASF GitHub Bot logged work on HDDS-1530:
Author: ASF GitHub Bot
Created on: 17/May/19 23:16
Start Date: 17/May/19 23:16
Worklog Time Spent: 10m
Work Description: xiaoyuyao commented on pull request #830: HDDS-1530.
Freon support big files larger than 2GB and add --bufferSize and
--validateWrites options.
URL: https://github.com/apache/hadoop/pull/830#discussion_r285313507
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/RandomKeyGenerator.java
##
@@ -622,7 +642,11 @@ public void run() {
try (Scope writeScope = GlobalTracer.get()
.buildSpan("writeKeyData")
.startActive(true)) {
- os.write(keyValue);
+ for (long nrRemaining = keySize - randomValue.length;
+nrRemaining > 0; nrRemaining -= bufferSize) {
+int curSize = (int)Math.min(bufferSize, nrRemaining);
+os.write(keyValueBuffer, 0, curSize);
Review comment:
Can we generate random string here and update the digest inline up to the
buffersize?
This way, we will not be writing the same buffer repeatedly that will be
cached in OS buffer/cache.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244426)
Time Spent: 40m (was: 0.5h)
> Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and
> "--validateWrites" options.
> --
>
> Key: HDDS-1530
> URL: https://issues.apache.org/jira/browse/HDDS-1530
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Xudong Cao
>Assignee: Xudong Cao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> *Current problems:*
> 1. Freon does not support big files larger than 2GB because it use an int
> type "keySize" parameter and also "keyValue" buffer size.
> 2. Freon allocates a entire buffer for each key at once, so if the key size
> is large and the concurrency is high, freon will report OOM exception
> frequently.
> 3. Freon lacks option such as "--validateWrites", thus users cannot manually
> specify that verification is required after writing.
> *Some solutions:*
> 1. Use a long type "keySize" parameter, make sure freon can support big
> files larger than 2GB.
> 2. Use a small buffer repeatedly than allocating the entire key-size buffer
> at once, the default buffer size is 4K and can be configured by "–bufferSize"
> parameter.
> 3. Add a "--validateWrites" option to Freon command line, users can provide
> this option to indicate that a validation is required after write.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong updated HDFS-14353: -- Attachment: HDFS-14353.009.patch > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14353.001.patch, HDFS-14353.002.patch, > HDFS-14353.003.patch, HDFS-14353.004.patch, HDFS-14353.005.patch, > HDFS-14353.006.patch, HDFS-14353.007.patch, HDFS-14353.008.patch, > HDFS-14353.009.patch, screenshot-1.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[
https://issues.apache.org/jira/browse/HDDS-1530?focusedWorklogId=244423=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244423
]
ASF GitHub Bot logged work on HDDS-1530:
Author: ASF GitHub Bot
Created on: 17/May/19 23:10
Start Date: 17/May/19 23:10
Worklog Time Spent: 10m
Work Description: xiaoyuyao commented on pull request #830: HDDS-1530.
Freon support big files larger than 2GB and add --bufferSize and
--validateWrites options.
URL: https://github.com/apache/hadoop/pull/830#discussion_r285312730
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/RandomKeyGenerator.java
##
@@ -228,8 +243,20 @@ public Void call() throws Exception {
init(freon.createOzoneConfiguration());
}
-keyValue =
-DFSUtil.string2Bytes(RandomStringUtils.randomAscii(keySize - 36));
+keyValueBuffer = DFSUtil.string2Bytes(
+RandomStringUtils.randomAscii(bufferSize));
+
+// Compute the common initial digest for all keys without their UUID
+if (validateWrites) {
+ commonInitialMD = DigestUtils.getDigest(DIGEST_ALGORITHM);
+ int uuidLength = UUID.randomUUID().toString().length();
+ keySize = Math.max(uuidLength, keySize);
+ for (long nrRemaining = keySize - uuidLength; nrRemaining > 0;
+ nrRemaining -= bufferSize) {
+int curSize = (int)Math.min(bufferSize, nrRemaining);
+commonInitialMD.update(keyValueBuffer, 0, curSize);
Review comment:
The keyValueBuffer is only generated randomly once before the loop.
Repeatedly calculate the same content won't change the digest here. Some of the
commonInitalMD.update() cycles can be saved here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244423)
Time Spent: 0.5h (was: 20m)
> Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and
> "--validateWrites" options.
> --
>
> Key: HDDS-1530
> URL: https://issues.apache.org/jira/browse/HDDS-1530
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Xudong Cao
>Assignee: Xudong Cao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> *Current problems:*
> 1. Freon does not support big files larger than 2GB because it use an int
> type "keySize" parameter and also "keyValue" buffer size.
> 2. Freon allocates a entire buffer for each key at once, so if the key size
> is large and the concurrency is high, freon will report OOM exception
> frequently.
> 3. Freon lacks option such as "--validateWrites", thus users cannot manually
> specify that verification is required after writing.
> *Some solutions:*
> 1. Use a long type "keySize" parameter, make sure freon can support big
> files larger than 2GB.
> 2. Use a small buffer repeatedly than allocating the entire key-size buffer
> at once, the default buffer size is 4K and can be configured by "–bufferSize"
> parameter.
> 3. Add a "--validateWrites" option to Freon command line, users can provide
> this option to indicate that a validation is required after write.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244413=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244413
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 17/May/19 22:58
Start Date: 17/May/19 22:58
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r285311016
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
+batchOperation);
+ } catch (IOException ex) {
+// During Adding to RocksDB batch entry got an exception.
+// We should terminate the OM.
+String message = "During flush to DB encountered error " +
+ex.getMessage();
+ExitUtil.terminate(1, message);
+ }
+});
+
+try {
+ omMetadataManager.getStore().commitBatchOperation(batchOperation);
+} catch (IOException ex) {
+ // During flush to rocksdb got an exception.
+ // We should terminate the OM.
+ String message = "During flush to DB encountered error " +
+ ex.getMessage();
+ ExitUtil.terminate(1, message);
+}
+
+int flushedTransactionsSize = readyBuffer.size();
+flushedTransactionCount.addAndGet(flushedTransactionsSize);
+flushIterations.incrementAndGet();
+
+LOG.info("Sync Iteration {} flushed transactions in this iteration{}",
+flushIterations.get(), flushedTransactionsSize);
+readyBuffer.clear();
+syncToDB = false;
+// TODO: update the last updated index in OzoneManagerStateMachine.
+ }
+}
+ }
+
+ /**
+ * Returns the flushed transaction count to OM DB.
+ * @return
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244412=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244412
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 17/May/19 22:57
Start Date: 17/May/19 22:57
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r285311016
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
+batchOperation);
+ } catch (IOException ex) {
+// During Adding to RocksDB batch entry got an exception.
+// We should terminate the OM.
+String message = "During flush to DB encountered error " +
+ex.getMessage();
+ExitUtil.terminate(1, message);
+ }
+});
+
+try {
+ omMetadataManager.getStore().commitBatchOperation(batchOperation);
+} catch (IOException ex) {
+ // During flush to rocksdb got an exception.
+ // We should terminate the OM.
+ String message = "During flush to DB encountered error " +
+ ex.getMessage();
+ ExitUtil.terminate(1, message);
+}
+
+int flushedTransactionsSize = readyBuffer.size();
+flushedTransactionCount.addAndGet(flushedTransactionsSize);
+flushIterations.incrementAndGet();
+
+LOG.info("Sync Iteration {} flushed transactions in this iteration{}",
+flushIterations.get(), flushedTransactionsSize);
+readyBuffer.clear();
+syncToDB = false;
+// TODO: update the last updated index in OzoneManagerStateMachine.
+ }
+}
+ }
+
+ /**
+ * Returns the flushed transaction count to OM DB.
+ * @return
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244410=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244410
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 17/May/19 22:54
Start Date: 17/May/19 22:54
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r285310539
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
Review comment:
My thought process of doing this way is, background flush will be
continuously happening if currentBuffer has any entries so that we don't wait
for the buffer to be full. So, that when a restart happens we don't need to
apply the transactions again to OM DB.
In the other way as suggested if we do, when the buffer is full, swap and
start flush, and keep further transactions adding to the currentBuffer, but if
flush is taking much time, and this buffer happened to be full, we should wait
until the first flush to be completed, and we might block all further
transactions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244410)
Time Spent: 2h 20m (was: 2h 10m)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> This Jira is created to implement DoubleBuffer in OzoneManager to flush
> transactions to OM DB.
>
> h2. Flushing Transactions to RocksDB:
> We propose using an implementation similar to the HDFS EditsDoubleBuffer. We
> shall flush RocksDB transactions in
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244411=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244411
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 17/May/19 22:54
Start Date: 17/May/19 22:54
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r285310539
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
Review comment:
My thought process of doing this way is, background flush will be
continuously happening if currentBuffer has any entries so that we don't wait
for the buffer to be full. So, that when a restart happens we don't need to
apply the transactions again to OM DB. And in this way we keep disks busy.
In the other way as suggested if we do, when the buffer is full, swap and
start flush, and keep further transactions adding to the currentBuffer, but if
flush is taking much time, and this buffer happened to be full, we should wait
until the first flush to be completed, and we might block all further
transactions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244411)
Time Spent: 2.5h (was: 2h 20m)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> This Jira is created to implement DoubleBuffer in OzoneManager to flush
> transactions to OM DB.
>
> h2. Flushing Transactions to RocksDB:
> We propose using an implementation similar to the HDFS EditsDoubleBuffer. We
> shall
[jira] [Commented] (HDDS-1521) Use more natural directory structure for ozone
[
https://issues.apache.org/jira/browse/HDDS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842768#comment-16842768
]
Eric Yang commented on HDDS-1521:
-
[~elek] {quote}We added the docs folder to the distribution tarbal as the
source of the information.{quote}
Ok. I see it in the ozone 0.4.0 tarball, but local build doesn't make that
directory. What is the mechanism to generate the html files?
> Use more natural directory structure for ozone
> --
>
> Key: HDDS-1521
> URL: https://issues.apache.org/jira/browse/HDDS-1521
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Elek, Marton
>Priority: Major
>
> The conversation of the current ozone directory structure is started here:
> HDDS-1458
> By [~eyang]
> {quote}The proposal is to generate test artifacts in
> ozone-[version]/share/ozone/tests for fault injection test framework in
> release binary tarball.
> {quote}
> By [~anu]:
> {quote}Let us drop the share and make it tests/blockade, and tests/smoketests
> etc. That way, all tests that we ship can be found easily. Otherwise I am +1
> on this change.
> {quote}
> By [~elek]
> {quote}I agree with Anu. I think It's time to revisit the current structure
> of ozone distribution tar files. The original structure is inherited from
> HADOOP-6255 which is defined for a project which includes multiple subproject
> (hadoop, yarn,...) and should support the creation of different RPM package
> from the same tar. I think for ozone we can improve the usability with
> reorganizing the dirs to a better structure.
> But I am fine to do it in a separated jira and keep it in the share/test
> until that to make progress.
> {quote}
> By [~eyang]
> {quote}The reason for HADOOP-6255 was more than just RPM packaging. The
> motivation behind the reorganization was to make a directory structure that
> can work as standalone tarball as well as follow the general guideline for
> [Filesystem Hierarchy
> Standard|https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard]. This
> was proposed because a several people saw the need to make the structure more
> unix like for sharing dependencies for larger eco-system to work. This is the
> reason that Hadoop HDFS, Mapreduce have good integration between projects to
> reduce shell script bootstrap cost. Earlier version of YARN did not follow
> the conventional wisdom and it was hard to integrate with rest of Hadoop,
> YARN community struggled on classpath issue for at least 2+ years and the
> time to hype YARN framework had already passed. Given there is a high
> probability that we want to make ozone as universal as possible for
> applications to integrate with us. It given us more incentive to make the
> structure as flexible as possible. This is only a advice from my own past
> scaring. There are no perfect solution, but the conventional wisdom usually
> have endure test of time and save energy.
> {quote}
> I propose the following method:
> # Start an independent discussion, let's don't mix this question with
> HDDS-1458 and don't block it.
> # Let's propose a more use friendly directory structure first
> # Let's ask [~eyang] to defined _technical_ problems regarding to the
> proposal.
> There are two different view here:
> # Technical limitations which were mentioned earlier
> # Theoretical questions (do we need to have the same structure for core
> Hadoop and Ozone packages)
> I propose to start the discussion with the first one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244401=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244401
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:40
Start Date: 17/May/19 22:40
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285305076
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -106,6 +142,40 @@ public void close() throws Exception {
}
+ @Override
+ public void addCacheEntry(CacheKey cacheKey,
+ CacheValue cacheValue) {
+// This will override the entry if there is already entry for this key.
+cache.put(cacheKey, cacheValue);
+ }
+
+
+ @Override
+ public void cleanupCache(long epoch) {
Review comment:
This is added because, in actual usage of classes, we get Table reference,
and they can simply call table.cleanupCache(). Added getCache() is used only
for testing, so declared it as not a public method and annotated with
VisibleForTesting. And usage of this will be in the code, once double-buffer
integration is completed.
Let me know if you still want to be removed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244401)
Time Spent: 10h 50m (was: 10h 40m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h 50m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244392=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244392 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 17/May/19 22:28 Start Date: 17/May/19 22:28 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-493617081 Thank You @arp7 for the review. I have addressed the review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244392) Time Spent: 10h 40m (was: 10.5h) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10h 40m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244391=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244391
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:28
Start Date: 17/May/19 22:28
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285306662
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/PartialTableCache.java
##
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import java.util.Iterator;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+/**
+ * Cache implementation for the table, this cache is partial cache, this will
+ * be cleaned up, after entries are flushed to DB.
+ */
+@Private
+@Evolving
+public class PartialTableCache implements TableCache
{
+
+ private final ConcurrentHashMap cache;
+ private final TreeSet> epochEntries;
+ private ExecutorService executorService;
+
+
+
+ public PartialTableCache() {
+cache = new ConcurrentHashMap<>();
+epochEntries = new TreeSet<>();
+// Created a singleThreadExecutor, so one cleanup will be running at a
+// time.
+executorService = Executors.newSingleThreadExecutor();
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244391)
Time Spent: 10.5h (was: 10h 20m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1065) OM and DN should persist SCM certificate as the trust root.
[ https://issues.apache.org/jira/browse/HDDS-1065?focusedWorklogId=244385=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244385 ] ASF GitHub Bot logged work on HDDS-1065: Author: ASF GitHub Bot Created on: 17/May/19 22:21 Start Date: 17/May/19 22:21 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #754: HDDS-1065. OM and DN should persist SCM certificate as the trust root. Contributed by Ajay Kumar. URL: https://github.com/apache/hadoop/pull/754#issuecomment-493615837 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 40 | https://github.com/apache/hadoop/pull/754 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/754 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-754/3/console | | versions | git=2.7.4 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244385) Time Spent: 1h 40m (was: 1.5h) > OM and DN should persist SCM certificate as the trust root. > --- > > Key: HDDS-1065 > URL: https://issues.apache.org/jira/browse/HDDS-1065 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > OM and DN should persist SCM certificate as the trust root. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244383=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244383 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 17/May/19 22:20 Start Date: 17/May/19 22:20 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r285305397 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java ## @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.hadoop.utils.db.cache; + +import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.classification.InterfaceStability.Evolving; + +/** + * Cache used for RocksDB tables. + * @param + * @param + */ + +@Private +@Evolving +public interface TableCache * @param */ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244383) Time Spent: 10h 20m (was: 10h 10m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10h 20m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14500) NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished
[
https://issues.apache.org/jira/browse/HDFS-14500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842674#comment-16842674
]
Hadoop QA commented on HDFS-14500:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 56 unchanged - 8 fixed = 56 total (was 64) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 59s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 22s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}166m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.aliasmap.TestSecureAliasMap |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
| | hadoop.hdfs.server.namenode.startupprogress.TestStartupProgress |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14500 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12969042/HDFS-14500.000.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5553a5ddef24 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 12c8161 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/26801/artifact/out/whitespace-eol.txt
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244381=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244381
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:18
Start Date: 17/May/19 22:18
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285305076
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -106,6 +142,40 @@ public void close() throws Exception {
}
+ @Override
+ public void addCacheEntry(CacheKey cacheKey,
+ CacheValue cacheValue) {
+// This will override the entry if there is already entry for this key.
+cache.put(cacheKey, cacheValue);
+ }
+
+
+ @Override
+ public void cleanupCache(long epoch) {
Review comment:
This is added because, in actual usage of classes, we get Table reference,
and they can simply call table.cleanupCache(). Added getCache() only for
testing, so declared it as not a public method and annotated with
VisibleForTesting. This will be used after double-buffer integration completed.
Let me know if you still want to be removed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244381)
Time Spent: 10h 10m (was: 10h)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h 10m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244378=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244378
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:15
Start Date: 17/May/19 22:15
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285304435
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -69,8 +79,34 @@ public boolean isEmpty() throws IOException {
return rawTable.isEmpty();
}
+ /**
+ * Returns the value mapped to the given key in byte array or returns null
+ * if the key is not found.
+ *
+ * Caller's of this method should use synchronization mechanism, when
+ * accessing. First it will check from cache, if it has entry return the
+ * value, otherwise get from the RocksDB table.
+ *
+ * @param key metadata key
+ * @return VALUE
+ * @throws IOException
+ */
@Override
public VALUE get(KEY key) throws IOException {
+// Here the metadata lock will guarantee that cache is not updated for same
+// key during get key.
+CacheValue< VALUE > cacheValue = cache.get(new CacheKey<>(key));
+if (cacheValue == null) {
+ // If no cache for the table or if it does not exist in cache get from
+ // RocksDB table.
+ return getFromTable(key);
+} else {
+ // We have a value in cache, return the value.
+ return cacheValue.getValue();
Review comment:
Yes actual value in CacheValue can be null.
**Implementation of CacheValue getValue**
public VALUE getValue() {
return value.orNull();
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244378)
Time Spent: 10h (was: 9h 50m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244362=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244362
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:14
Start Date: 17/May/19 22:14
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285304280
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -31,13 +37,16 @@
*/
public class TypedTable implements Table {
- private Table rawTable;
+ private final Table rawTable;
+
+ private final CodecRegistry codecRegistry;
- private CodecRegistry codecRegistry;
+ private final Class keyType;
Review comment:
Used by TableIterator, to deserialize actual table byte[] data to particular
key value format it iwill be used.
Note: This is not added by this change, only converted them to final.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244362)
Time Spent: 9h 50m (was: 9h 40m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 50m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1422) Exception during DataNode shutdown
[
https://issues.apache.org/jira/browse/HDDS-1422?focusedWorklogId=244360=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244360
]
ASF GitHub Bot logged work on HDDS-1422:
Author: ASF GitHub Bot
Created on: 17/May/19 22:09
Start Date: 17/May/19 22:09
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #725:
HDDS-1422. Exception during DataNode shutdown. Contributed by Arpit A…
URL: https://github.com/apache/hadoop/pull/725#discussion_r285303293
##
File path: phantomjsdriver.log
##
@@ -0,0 +1,6 @@
+[INFO - 2019-05-13T20:32:17.166Z] GhostDriver - Main - running on port 3369
+[INFO - 2019-05-13T20:32:17.424Z] Session
[332af4c0-75be-11e9-ae47-adcdd47c587b] - page.settings -
{"XSSAuditingEnabled":false,"javascriptCanCloseWindows":true,"javascriptCanOpenWindows":true,"javascriptEnabled":true,"loadImages":true,"localToRemoteUrlAccessEnabled":false,"userAgent":"Mozilla/5.0
(Macintosh; Intel Mac OS X) AppleWebKit/538.1 (KHTML, like Gecko)
PhantomJS/2.1.1 Safari/538.1","webSecurityEnabled":true}
+[INFO - 2019-05-13T20:32:17.424Z] Session
[332af4c0-75be-11e9-ae47-adcdd47c587b] - page.customHeaders: - {}
+[INFO - 2019-05-13T20:32:17.425Z] Session
[332af4c0-75be-11e9-ae47-adcdd47c587b] - Session.negotiatedCapabilities -
{"browserName":"phantomjs","version":"2.1.1","driverName":"ghostdriver","driverVersion":"1.2.0","platform":"mac-unknown-64bit","javascriptEnabled":true,"takesScreenshot":true,"handlesAlerts":false,"databaseEnabled":false,"locationContextEnabled":false,"applicationCacheEnabled":false,"browserConnectionEnabled":false,"cssSelectorsEnabled":true,"webStorageEnabled":false,"rotatable":false,"acceptSslCerts":false,"nativeEvents":true,"proxy":{"proxyType":"direct"}}
Review comment:
I think a file is added by mistake.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244360)
Time Spent: 1h 50m (was: 1h 40m)
> Exception during DataNode shutdown
> --
>
> Key: HDDS-1422
> URL: https://issues.apache.org/jira/browse/HDDS-1422
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> The following exception during DN shutdown should be avoided, as it adds
> noise to the logs and is not a real issue.
> {code}
> 2019-04-10 17:48:27,307 WARN volume.VolumeSet
> (VolumeSet.java:getNodeReport(476)) - Failed to get scmUsed and remaining for
> container storage location
> /Users/agarwal/src/hadoop/hadoop-ozone/integration-test/target/test/data/MiniOzoneClusterImpl-f4d89966-146a-4690-8841-36af1993522f/datanode-17/data/containers
> java.io.IOException: Volume Usage thread is not running. This error is
> usually seen during DataNode shutdown.
> at
> org.apache.hadoop.ozone.container.common.volume.VolumeInfo.getScmUsed(VolumeInfo.java:119)
> at
> org.apache.hadoop.ozone.container.common.volume.VolumeSet.getNodeReport(VolumeSet.java:472)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.getNodeReport(OzoneContainer.java:238)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.RegisterEndpointTask.call(RegisterEndpointTask.java:115)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.RegisterEndpointTask.call(RegisterEndpointTask.java:47)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244355=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244355
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:05
Start Date: 17/May/19 22:05
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285301711
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -69,8 +79,34 @@ public boolean isEmpty() throws IOException {
return rawTable.isEmpty();
}
+ /**
+ * Returns the value mapped to the given key in byte array or returns null
+ * if the key is not found.
+ *
+ * Caller's of this method should use synchronization mechanism, when
+ * accessing. First it will check from cache, if it has entry return the
+ * value, otherwise get from the RocksDB table.
+ *
+ * @param key metadata key
+ * @return VALUE
+ * @throws IOException
+ */
@Override
public VALUE get(KEY key) throws IOException {
+// Here the metadata lock will guarantee that cache is not updated for same
+// key during get key.
+CacheValue< VALUE > cacheValue = cache.get(new CacheKey<>(key));
+if (cacheValue == null) {
+ // If no cache for the table or if it does not exist in cache get from
+ // RocksDB table.
+ return getFromTable(key);
+} else {
+ // We have a value in cache, return the value.
+ return cacheValue.getValue();
Review comment:
cacheValue.getValue() can be a null to indicate deletion, correct?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244355)
Time Spent: 9h 20m (was: 9h 10m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244357=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244357
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:05
Start Date: 17/May/19 22:05
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285302353
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/PartialTableCache.java
##
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import java.util.Iterator;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+/**
+ * Cache implementation for the table, this cache is partial cache, this will
+ * be cleaned up, after entries are flushed to DB.
+ */
+@Private
+@Evolving
+public class PartialTableCache implements TableCache
{
+
+ private final ConcurrentHashMap cache;
+ private final TreeSet> epochEntries;
+ private ExecutorService executorService;
+
+
+
+ public PartialTableCache() {
+cache = new ConcurrentHashMap<>();
+epochEntries = new TreeSet<>();
+// Created a singleThreadExecutor, so one cleanup will be running at a
+// time.
+executorService = Executors.newSingleThreadExecutor();
Review comment:
Can you use ThreadFactory here to give the executor thread a descriptive
name?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244357)
Time Spent: 9h 40m (was: 9.5h)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244356=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244356
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:05
Start Date: 17/May/19 22:05
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285301960
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -106,6 +142,40 @@ public void close() throws Exception {
}
+ @Override
+ public void addCacheEntry(CacheKey cacheKey,
+ CacheValue cacheValue) {
+// This will override the entry if there is already entry for this key.
+cache.put(cacheKey, cacheValue);
+ }
+
+
+ @Override
+ public void cleanupCache(long epoch) {
Review comment:
We can remove this function I think. Caller can do
`table.getCache().cleanup()`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244356)
Time Spent: 9.5h (was: 9h 20m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244358=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244358 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 17/May/19 22:05 Start Date: 17/May/19 22:05 Worklog Time Spent: 10m Work Description: arp7 commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r285302472 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java ## @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.hadoop.utils.db.cache; + +import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.classification.InterfaceStability.Evolving; + +/** + * Cache used for RocksDB tables. + * @param + * @param + */ + +@Private +@Evolving +public interface TableCache OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 9h 40m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244354=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244354
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:05
Start Date: 17/May/19 22:05
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285301379
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -31,13 +37,16 @@
*/
public class TypedTable implements Table {
- private Table rawTable;
+ private final Table rawTable;
+
+ private final CodecRegistry codecRegistry;
- private CodecRegistry codecRegistry;
+ private final Class keyType;
Review comment:
@bharatviswa504 why do we need the keyType and valueType fields?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244354)
Time Spent: 9h 20m (was: 9h 10m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842625#comment-16842625
]
Arpit Agarwal commented on HDDS-1458:
-
I was able to get the tests to run with the v7 patch. However most tests failed
with the following error:
{code}
> assert exit_code == 0, "freon run failed with output=[%s]" % output
E AssertionError: freon run failed with output=[the input device is not a
TTY]
E assert 1 == 0
{code}
Not sure if freon has some assumption about stdin being a tty.
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Attachments: HDDS-1458.001.patch, HDDS-1458.002.patch,
> HDDS-1458.003.patch, HDDS-1458.004.patch, HDDS-1458.005.patch,
> HDDS-1458.006.patch, HDDS-1458.007.patch, HDDS-1458.008.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1440) Convert all MPU related operations to HA model
[ https://issues.apache.org/jira/browse/HDDS-1440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-1440: Labels: (was: pull-request-available) > Convert all MPU related operations to HA model > -- > > Key: HDDS-1440 > URL: https://issues.apache.org/jira/browse/HDDS-1440 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Fix For: 0.4.1 > > > In this jira, we shall convert all OM related operations to OM HA model, > which is a 2 step. > # StartTransaction, where we validate request and check for any errors and > return the response. > # ApplyTransaction, where original OM request will have a response which > needs to be applied to OM DB. This step is just to apply response to Om DB. > In this way, all requests which are failed with like volume not found or some > conditions which i have not satisfied like when deleting volume should be > empty, these all will be executed during startTransaction, and if it fails > these requests will not be written to raft log also. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1400) Convert all OM Key related operations to HA model
[ https://issues.apache.org/jira/browse/HDDS-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-1400: Labels: (was: pull-request-available) > Convert all OM Key related operations to HA model > - > > Key: HDDS-1400 > URL: https://issues.apache.org/jira/browse/HDDS-1400 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Fix For: 0.4.1 > > Time Spent: 20m > Remaining Estimate: 0h > > In this jira, we shall convert all OM related operations to OM HA model, > which is a 2 step. > # StartTransaction, where we validate request and check for any errors and > return the response. > # ApplyTransaction, where original OM request will have a response which > needs to be applied to OM DB. This step is just to apply response to Om DB. > In this way, all requests which are failed with like volume not found or some > conditions which are not satisfied like when Key not found during rename, > these all will be executed during startTransaction, and if it fails these > requests will not be written to raft log also. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1347) In OM HA getS3Secret call Should happen only leader OM
[ https://issues.apache.org/jira/browse/HDDS-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-1347: Labels: (was: pull-request-available) > In OM HA getS3Secret call Should happen only leader OM > -- > > Key: HDDS-1347 > URL: https://issues.apache.org/jira/browse/HDDS-1347 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > In Om HA getS3Secret should happen only leader OM. > > > The reason is similar to initiateMultipartUpload. For more info refer > HDDS-1319 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842598#comment-16842598
]
Hadoop QA commented on HDDS-1458:
-
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://builds.apache.org/job/PreCommit-HDDS-Build/2698/console in case of
problems.
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Attachments: HDDS-1458.001.patch, HDDS-1458.002.patch,
> HDDS-1458.003.patch, HDDS-1458.004.patch, HDDS-1458.005.patch,
> HDDS-1458.006.patch, HDDS-1458.007.patch, HDDS-1458.008.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1517) AllocateBlock call fails with ContainerNotFoundException
[ https://issues.apache.org/jira/browse/HDDS-1517?focusedWorklogId=244292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244292 ] ASF GitHub Bot logged work on HDDS-1517: Author: ASF GitHub Bot Created on: 17/May/19 20:53 Start Date: 17/May/19 20:53 Worklog Time Spent: 10m Work Description: jnp commented on pull request #826: HDDS-1517. AllocateBlock call fails with ContainerNotFoundException. URL: https://github.com/apache/hadoop/pull/826#discussion_r285285507 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerStateManager.java ## @@ -296,10 +296,10 @@ ContainerInfo allocateContainer( .setReplicationFactor(pipeline.getFactor()) .setReplicationType(pipeline.getType()) .build(); -pipelineManager.addContainerToPipeline(pipeline.getId(), Review comment: Should we also add a comment that this method must be called with lock held on the pipeline? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244292) > AllocateBlock call fails with ContainerNotFoundException > > > Key: HDDS-1517 > URL: https://issues.apache.org/jira/browse/HDDS-1517 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.5.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Attachments: HDDS-1517.000.patch > > Time Spent: 1h > Remaining Estimate: 0h > > In allocateContainer call, the container is first added to pipelineStateMap > and then added to container cache. If two allocate blocks execute > concurrently, it might happen that one find the container to exist in the > pipelineStateMap but the container is yet to be updated in the container > cache, hence failing with CONTAINER_NOT_FOUND exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1517) AllocateBlock call fails with ContainerNotFoundException
[ https://issues.apache.org/jira/browse/HDDS-1517?focusedWorklogId=244290=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244290 ] ASF GitHub Bot logged work on HDDS-1517: Author: ASF GitHub Bot Created on: 17/May/19 20:53 Start Date: 17/May/19 20:53 Worklog Time Spent: 10m Work Description: jnp commented on pull request #826: HDDS-1517. AllocateBlock call fails with ContainerNotFoundException. URL: https://github.com/apache/hadoop/pull/826#discussion_r285283814 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerStateManager.java ## @@ -296,10 +296,10 @@ ContainerInfo allocateContainer( .setReplicationFactor(pipeline.getFactor()) .setReplicationType(pipeline.getType()) .build(); -pipelineManager.addContainerToPipeline(pipeline.getId(), Review comment: I see that this method (allocateContainer) is also called from places where lock on the pipeline is not held. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244290) Time Spent: 50m (was: 40m) > AllocateBlock call fails with ContainerNotFoundException > > > Key: HDDS-1517 > URL: https://issues.apache.org/jira/browse/HDDS-1517 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.5.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Attachments: HDDS-1517.000.patch > > Time Spent: 50m > Remaining Estimate: 0h > > In allocateContainer call, the container is first added to pipelineStateMap > and then added to container cache. If two allocate blocks execute > concurrently, it might happen that one find the container to exist in the > pipelineStateMap but the container is yet to be updated in the container > cache, hence failing with CONTAINER_NOT_FOUND exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1517) AllocateBlock call fails with ContainerNotFoundException
[
https://issues.apache.org/jira/browse/HDDS-1517?focusedWorklogId=244291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244291
]
ASF GitHub Bot logged work on HDDS-1517:
Author: ASF GitHub Bot
Created on: 17/May/19 20:53
Start Date: 17/May/19 20:53
Worklog Time Spent: 10m
Work Description: jnp commented on pull request #826: HDDS-1517.
AllocateBlock call fails with ContainerNotFoundException.
URL: https://github.com/apache/hadoop/pull/826#discussion_r285284257
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/SCMContainerManager.java
##
@@ -386,18 +386,17 @@ public ContainerInfo getMatchingContainer(final long
sizeRequired,
public ContainerInfo getMatchingContainer(final long sizeRequired,
String owner, Pipeline pipeline, List excludedContainers) {
+NavigableSet containerIDs;
try {
- //TODO: #CLUTIL See if lock is required here
- NavigableSet containerIDs =
- pipelineManager.getContainersInPipeline(pipeline.getId());
+ synchronized (pipeline) {
+//TODO: #CLUTIL See if lock is required here
+containerIDs =
+pipelineManager.getContainersInPipeline(pipeline.getId());
- containerIDs = getContainersForOwner(containerIDs, owner);
- if (containerIDs.size() < numContainerPerOwnerInPipeline) {
-synchronized (pipeline) {
+containerIDs = getContainersForOwner(containerIDs, owner);
+if (containerIDs.size() < numContainerPerOwnerInPipeline) {
// TODO: #CLUTIL Maybe we can add selection logic inside synchronized
// as well
- containerIDs = getContainersForOwner(
- pipelineManager.getContainersInPipeline(pipeline.getId()),
owner);
if (containerIDs.size() < numContainerPerOwnerInPipeline) {
ContainerInfo containerInfo =
containerStateManager.allocateContainer(pipelineManager, owner,
Review comment:
There is one more place where containerStateManager#allocateContainer is
being called from, where pipeline lock is not acquired.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244291)
Time Spent: 1h (was: 50m)
> AllocateBlock call fails with ContainerNotFoundException
>
>
> Key: HDDS-1517
> URL: https://issues.apache.org/jira/browse/HDDS-1517
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.5.0
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Attachments: HDDS-1517.000.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> In allocateContainer call, the container is first added to pipelineStateMap
> and then added to container cache. If two allocate blocks execute
> concurrently, it might happen that one find the container to exist in the
> pipelineStateMap but the container is yet to be updated in the container
> cache, hence failing with CONTAINER_NOT_FOUND exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HDDS-1458:
Attachment: HDDS-1458.008.patch
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Attachments: HDDS-1458.001.patch, HDDS-1458.002.patch,
> HDDS-1458.003.patch, HDDS-1458.004.patch, HDDS-1458.005.patch,
> HDDS-1458.006.patch, HDDS-1458.007.patch, HDDS-1458.008.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-700) Support rack awared node placement policy based on network topology
[
https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842560#comment-16842560
]
Siddharth Wagle edited comment on HDDS-700 at 5/17/19 8:33 PM:
---
[~Sammi] If the topology is modeled as "*/d1/r1/ng/n*", will the cross rack
logic still work for replicas 3 and above? The current logic will choose the
second replica on the same NG but will the third replica correctly land on a
separate rack and not just separate NG?
was (Author: swagle):
[~Sammi] If the topology is modeled as "*/d1/r1/ng/n*", will the cross rack
logic still work for replcas 3 and above? The current logic will choose the
second replica on the same NG but will the thrid replica correctly land on a
separate rack and not just NG?
> Support rack awared node placement policy based on network topology
> ---
>
> Key: HDDS-700
> URL: https://issues.apache.org/jira/browse/HDDS-700
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Sammi Chen
>Priority: Major
> Attachments: HDDS-700.01.patch
>
>
> Implement a new container placement policy implementation based datanode's
> network topology. It follows the same rule as HDFS.
> By default with 3 replica, two replica will be on the same rack, the third
> replica and all the remaining replicas will be on different racks.
>
> {color:#808080} {color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-700) Support rack awared node placement policy based on network topology
[
https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842560#comment-16842560
]
Siddharth Wagle commented on HDDS-700:
--
[~Sammi] If the topology is modeled as "*/d1/r1/ng/n*", will the cross rack
logic still work for replcas 3 and above? The current logic will choose the
second replica on the same NG but will the thrid replica correctly land on a
separate rack and not just NG?
> Support rack awared node placement policy based on network topology
> ---
>
> Key: HDDS-700
> URL: https://issues.apache.org/jira/browse/HDDS-700
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Sammi Chen
>Priority: Major
> Attachments: HDDS-700.01.patch
>
>
> Implement a new container placement policy implementation based datanode's
> network topology. It follows the same rule as HDFS.
> By default with 3 replica, two replica will be on the same rack, the third
> replica and all the remaining replicas will be on different racks.
>
> {color:#808080} {color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14501) BenchmarkThroughput.writeFile hangs with misconfigured BUFFER_SIZE
John Doe created HDFS-14501:
---
Summary: BenchmarkThroughput.writeFile hangs with misconfigured
BUFFER_SIZE
Key: HDFS-14501
URL: https://issues.apache.org/jira/browse/HDFS-14501
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: John Doe
When the configuration file is corrupted, reading BUFFER_SIZE from corrupted
conf can return 0.
The "for" loop in BenchmarkThroughput.writeLocalFile function hangs endlessly.
Here is the code snippet.
{code:java}
BUFFER_SIZE = conf.getInt("dfsthroughput.buffer.size", 4 * 1024);
private Path writeFile(FileSystem fs,
String name,
Configuration conf,
long total
) throws IOException {
Path f = dir.getLocalPathForWrite(name, total, conf);
System.out.print("Writing " + name);
resetMeasurements();
OutputStream out = fs.create(f);
byte[] data = new byte[BUFFER_SIZE];
for(long size = 0; size < total; size += BUFFER_SIZE) { //Bug!
out.write(data);
}
out.close();
printMeasurements();
return f;
}
{code}
This configuration error also affects HDFS-13513, HDFS-13514,
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14500) NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished
[
https://issues.apache.org/jira/browse/HDFS-14500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842412#comment-16842412
]
Erik Krogen commented on HDFS-14500:
Attached v000 patch which fixes this issue by turning the phase-specific steps
into no-ops if that phase is completed, just like how they are no-ops once all
of startup completes. I updated the unit test accordingly.
> NameNode StartupProgress continues to report edit log segments after the
> LOADING_EDITS phase is finished
>
>
> Key: HDFS-14500
> URL: https://issues.apache.org/jira/browse/HDFS-14500
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0, 2.9.2, 3.0.3, 2.8.5, 3.1.2
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14500.000.patch
>
>
> When testing out a cluster with the edit log tailing fast path feature
> enabled (HDFS-13150), an unrelated issue caused the NameNode to remain in
> safe mode for an extended period of time, preventing the NameNode from fully
> completing its startup sequence. We noticed that the Startup Progress web UI
> displayed many edit log segments (millions of them).
> I traced this problem back to {{StartupProgress}}. Within
> {{FSEditLogLoader}}, the loader continually tries to update the startup
> progress with a new {{Step}} any time that it loads edits. Per the Javadoc
> for {{StartupProgress}}, this should be a no-op once startup is completed:
> {code:title=StartupProgress.java}
> * After startup completes, the tracked data is frozen. Any subsequent
> updates
> * or counter increments are no-ops.
> {code}
> However, {{StartupProgress}} only implements that logic once the _entire_
> startup sequence has been completed. When {{FSEditLogLoader}} calls
> {{addStep()}}, it adds it into the {{LOADING_EDITS}} phase:
> {code:title=FSEditLogLoader.java}
> StartupProgress prog = NameNode.getStartupProgress();
> Step step = createStartupProgressStep(edits);
> prog.beginStep(Phase.LOADING_EDITS, step);
> {code}
> This phase, in our case, ended long before, so it is nonsensical to continue
> to add steps to it. I believe it is a bug that {{StartupProgress}} accepts
> such steps instead of ignoring them; once a phase is complete, it should no
> longer change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14500) NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished
[
https://issues.apache.org/jira/browse/HDFS-14500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-14500:
---
Attachment: HDFS-14500.000.patch
> NameNode StartupProgress continues to report edit log segments after the
> LOADING_EDITS phase is finished
>
>
> Key: HDFS-14500
> URL: https://issues.apache.org/jira/browse/HDFS-14500
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0, 2.9.2, 3.0.3, 2.8.5, 3.1.2
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14500.000.patch
>
>
> When testing out a cluster with the edit log tailing fast path feature
> enabled (HDFS-13150), an unrelated issue caused the NameNode to remain in
> safe mode for an extended period of time, preventing the NameNode from fully
> completing its startup sequence. We noticed that the Startup Progress web UI
> displayed many edit log segments (millions of them).
> I traced this problem back to {{StartupProgress}}. Within
> {{FSEditLogLoader}}, the loader continually tries to update the startup
> progress with a new {{Step}} any time that it loads edits. Per the Javadoc
> for {{StartupProgress}}, this should be a no-op once startup is completed:
> {code:title=StartupProgress.java}
> * After startup completes, the tracked data is frozen. Any subsequent
> updates
> * or counter increments are no-ops.
> {code}
> However, {{StartupProgress}} only implements that logic once the _entire_
> startup sequence has been completed. When {{FSEditLogLoader}} calls
> {{addStep()}}, it adds it into the {{LOADING_EDITS}} phase:
> {code:title=FSEditLogLoader.java}
> StartupProgress prog = NameNode.getStartupProgress();
> Step step = createStartupProgressStep(edits);
> prog.beginStep(Phase.LOADING_EDITS, step);
> {code}
> This phase, in our case, ended long before, so it is nonsensical to continue
> to add steps to it. I believe it is a bug that {{StartupProgress}} accepts
> such steps instead of ignoring them; once a phase is complete, it should no
> longer change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14500) NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished
[
https://issues.apache.org/jira/browse/HDFS-14500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-14500:
---
Status: Patch Available (was: In Progress)
> NameNode StartupProgress continues to report edit log segments after the
> LOADING_EDITS phase is finished
>
>
> Key: HDFS-14500
> URL: https://issues.apache.org/jira/browse/HDFS-14500
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.1.2, 2.8.5, 3.0.3, 2.9.2, 3.2.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14500.000.patch
>
>
> When testing out a cluster with the edit log tailing fast path feature
> enabled (HDFS-13150), an unrelated issue caused the NameNode to remain in
> safe mode for an extended period of time, preventing the NameNode from fully
> completing its startup sequence. We noticed that the Startup Progress web UI
> displayed many edit log segments (millions of them).
> I traced this problem back to {{StartupProgress}}. Within
> {{FSEditLogLoader}}, the loader continually tries to update the startup
> progress with a new {{Step}} any time that it loads edits. Per the Javadoc
> for {{StartupProgress}}, this should be a no-op once startup is completed:
> {code:title=StartupProgress.java}
> * After startup completes, the tracked data is frozen. Any subsequent
> updates
> * or counter increments are no-ops.
> {code}
> However, {{StartupProgress}} only implements that logic once the _entire_
> startup sequence has been completed. When {{FSEditLogLoader}} calls
> {{addStep()}}, it adds it into the {{LOADING_EDITS}} phase:
> {code:title=FSEditLogLoader.java}
> StartupProgress prog = NameNode.getStartupProgress();
> Step step = createStartupProgressStep(edits);
> prog.beginStep(Phase.LOADING_EDITS, step);
> {code}
> This phase, in our case, ended long before, so it is nonsensical to continue
> to add steps to it. I believe it is a bug that {{StartupProgress}} accepts
> such steps instead of ignoring them; once a phase is complete, it should no
> longer change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-14500) NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished
[
https://issues.apache.org/jira/browse/HDFS-14500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-14500 started by Erik Krogen.
--
> NameNode StartupProgress continues to report edit log segments after the
> LOADING_EDITS phase is finished
>
>
> Key: HDFS-14500
> URL: https://issues.apache.org/jira/browse/HDFS-14500
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0, 2.9.2, 3.0.3, 2.8.5, 3.1.2
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
>
> When testing out a cluster with the edit log tailing fast path feature
> enabled (HDFS-13150), an unrelated issue caused the NameNode to remain in
> safe mode for an extended period of time, preventing the NameNode from fully
> completing its startup sequence. We noticed that the Startup Progress web UI
> displayed many edit log segments (millions of them).
> I traced this problem back to {{StartupProgress}}. Within
> {{FSEditLogLoader}}, the loader continually tries to update the startup
> progress with a new {{Step}} any time that it loads edits. Per the Javadoc
> for {{StartupProgress}}, this should be a no-op once startup is completed:
> {code:title=StartupProgress.java}
> * After startup completes, the tracked data is frozen. Any subsequent
> updates
> * or counter increments are no-ops.
> {code}
> However, {{StartupProgress}} only implements that logic once the _entire_
> startup sequence has been completed. When {{FSEditLogLoader}} calls
> {{addStep()}}, it adds it into the {{LOADING_EDITS}} phase:
> {code:title=FSEditLogLoader.java}
> StartupProgress prog = NameNode.getStartupProgress();
> Step step = createStartupProgressStep(edits);
> prog.beginStep(Phase.LOADING_EDITS, step);
> {code}
> This phase, in our case, ended long before, so it is nonsensical to continue
> to add steps to it. I believe it is a bug that {{StartupProgress}} accepts
> such steps instead of ignoring them; once a phase is complete, it should no
> longer change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14500) NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished
Erik Krogen created HDFS-14500:
--
Summary: NameNode StartupProgress continues to report edit log
segments after the LOADING_EDITS phase is finished
Key: HDFS-14500
URL: https://issues.apache.org/jira/browse/HDFS-14500
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 3.1.2, 2.8.5, 3.0.3, 2.9.2, 3.2.0
Reporter: Erik Krogen
Assignee: Erik Krogen
When testing out a cluster with the edit log tailing fast path feature enabled
(HDFS-13150), an unrelated issue caused the NameNode to remain in safe mode for
an extended period of time, preventing the NameNode from fully completing its
startup sequence. We noticed that the Startup Progress web UI displayed many
edit log segments (millions of them).
I traced this problem back to {{StartupProgress}}. Within {{FSEditLogLoader}},
the loader continually tries to update the startup progress with a new {{Step}}
any time that it loads edits. Per the Javadoc for {{StartupProgress}}, this
should be a no-op once startup is completed:
{code:title=StartupProgress.java}
* After startup completes, the tracked data is frozen. Any subsequent updates
* or counter increments are no-ops.
{code}
However, {{StartupProgress}} only implements that logic once the _entire_
startup sequence has been completed. When {{FSEditLogLoader}} calls
{{addStep()}}, it adds it into the {{LOADING_EDITS}} phase:
{code:title=FSEditLogLoader.java}
StartupProgress prog = NameNode.getStartupProgress();
Step step = createStartupProgressStep(edits);
prog.beginStep(Phase.LOADING_EDITS, step);
{code}
This phase, in our case, ended long before, so it is nonsensical to continue to
add steps to it. I believe it is a bug that {{StartupProgress}} accepts such
steps instead of ignoring them; once a phase is complete, it should no longer
change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842408#comment-16842408
]
Eric Yang commented on HDDS-1458:
-
[~elek] {quote} I prefer to do it one step. Better to add something when it can
be tested how does it work. If it's not possible now, because there are now
tests yet, let's modify only the blockade part without the disk tests. I assume
there will be a new jira which explains the plan with the disk based
fault-injection tests.{quote}
Ok, I'll removed disk-tests submodules from this issue. Disk related test is
opened as HDDS-1554.
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Attachments: HDDS-1458.001.patch, HDDS-1458.002.patch,
> HDDS-1458.003.patch, HDDS-1458.004.patch, HDDS-1458.005.patch,
> HDDS-1458.006.patch, HDDS-1458.007.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1554) Create disk tests for fault injection test
Eric Yang created HDDS-1554: --- Summary: Create disk tests for fault injection test Key: HDDS-1554 URL: https://issues.apache.org/jira/browse/HDDS-1554 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: build Reporter: Eric Yang The current plan for fault injection disk tests are: # Scenario 1 - Read/Write test ## Run docker-compose to bring up a cluster ## Initialize scm and om ## Upload data to Ozone cluster ## Verify data is correct ## Shutdown cluster # Scenario 2 - Read/Only test ## Repeat Scenario 1 ## Mount data disk as read only ## Try to write data to Ozone cluster ## Validate error message is correct ## Shutdown cluster # Scenario 3 - Corruption test ## Repeat Scenario 2 ## Shutdown cluster ## Modify data disk data ## Restart cluster ## Validate error message for read from corrupted data ## Validate error message for write to corrupted volume -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244215=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244215 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 17/May/19 19:00 Start Date: 17/May/19 19:00 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-493563512 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 58 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 84 | Maven dependency ordering for branch | | +1 | mvninstall | 576 | trunk passed | | +1 | compile | 238 | trunk passed | | +1 | checkstyle | 61 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 1015 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 149 | trunk passed | | 0 | spotbugs | 288 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 534 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for patch | | +1 | mvninstall | 533 | the patch passed | | +1 | compile | 256 | the patch passed | | +1 | javac | 256 | the patch passed | | +1 | checkstyle | 68 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 774 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 153 | the patch passed | | +1 | findbugs | 584 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 188 | hadoop-hdds in the patch failed. | | -1 | unit | 1100 | hadoop-ozone in the patch failed. | | +1 | asflicense | 36 | The patch does not generate ASF License warnings. | | | | 6581 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestBlockOutputStream | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | | hadoop.ozone.client.rpc.TestKeyInputStream | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-798/9/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/798 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux deac2877b50b 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 4cb3da6 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/9/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/9/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/9/testReport/ | | Max. process+thread count | 3554 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/server-scm hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/9/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244215) Time Spent: 9h 10m (was: 9h) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 >
[jira] [Commented] (HDDS-1521) Use more natural directory structure for ozone
[
https://issues.apache.org/jira/browse/HDDS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842387#comment-16842387
]
Elek, Marton commented on HDDS-1521:
Thanks the answer [~eyang].
1. Agree, maintaining different structure for deb/rpm is an additional work but
we may need it. I don't know yet. There are pros and cons.
2. classpath issue is discussed in HDDS-1520 (we can't switch back to the old
approach IMHO)
3. We added the docs folder to the distribution tarbal as the source of the
information.
> Use more natural directory structure for ozone
> --
>
> Key: HDDS-1521
> URL: https://issues.apache.org/jira/browse/HDDS-1521
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Elek, Marton
>Priority: Major
>
> The conversation of the current ozone directory structure is started here:
> HDDS-1458
> By [~eyang]
> {quote}The proposal is to generate test artifacts in
> ozone-[version]/share/ozone/tests for fault injection test framework in
> release binary tarball.
> {quote}
> By [~anu]:
> {quote}Let us drop the share and make it tests/blockade, and tests/smoketests
> etc. That way, all tests that we ship can be found easily. Otherwise I am +1
> on this change.
> {quote}
> By [~elek]
> {quote}I agree with Anu. I think It's time to revisit the current structure
> of ozone distribution tar files. The original structure is inherited from
> HADOOP-6255 which is defined for a project which includes multiple subproject
> (hadoop, yarn,...) and should support the creation of different RPM package
> from the same tar. I think for ozone we can improve the usability with
> reorganizing the dirs to a better structure.
> But I am fine to do it in a separated jira and keep it in the share/test
> until that to make progress.
> {quote}
> By [~eyang]
> {quote}The reason for HADOOP-6255 was more than just RPM packaging. The
> motivation behind the reorganization was to make a directory structure that
> can work as standalone tarball as well as follow the general guideline for
> [Filesystem Hierarchy
> Standard|https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard]. This
> was proposed because a several people saw the need to make the structure more
> unix like for sharing dependencies for larger eco-system to work. This is the
> reason that Hadoop HDFS, Mapreduce have good integration between projects to
> reduce shell script bootstrap cost. Earlier version of YARN did not follow
> the conventional wisdom and it was hard to integrate with rest of Hadoop,
> YARN community struggled on classpath issue for at least 2+ years and the
> time to hype YARN framework had already passed. Given there is a high
> probability that we want to make ozone as universal as possible for
> applications to integrate with us. It given us more incentive to make the
> structure as flexible as possible. This is only a advice from my own past
> scaring. There are no perfect solution, but the conventional wisdom usually
> have endure test of time and save energy.
> {quote}
> I propose the following method:
> # Start an independent discussion, let's don't mix this question with
> HDDS-1458 and don't block it.
> # Let's propose a more use friendly directory structure first
> # Let's ask [~eyang] to defined _technical_ problems regarding to the
> proposal.
> There are two different view here:
> # Technical limitations which were mentioned earlier
> # Theoretical questions (do we need to have the same structure for core
> Hadoop and Ozone packages)
> I propose to start the discussion with the first one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1538) Update ozone protobuf message for ACLs
[
https://issues.apache.org/jira/browse/HDDS-1538?focusedWorklogId=244198=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244198
]
ASF GitHub Bot logged work on HDDS-1538:
Author: ASF GitHub Bot
Created on: 17/May/19 18:50
Start Date: 17/May/19 18:50
Worklog Time Spent: 10m
Work Description: xiaoyuyao commented on pull request #828: HDDS-1538.
Update ozone protobuf message for ACLs. Contributed by Ajay Kumar.
URL: https://github.com/apache/hadoop/pull/828#discussion_r285247846
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OzoneAcl.java
##
@@ -47,16 +52,37 @@ public OzoneAcl() {
*
* @param type - Type
* @param name - Name of user
- * @param rights - Rights
+ * @param acl - Rights
*/
- public OzoneAcl(OzoneACLType type, String name, OzoneACLRights rights) {
+ public OzoneAcl(ACLIdentityType type, String name, ACLType acl) {
this.name = name;
-this.rights = rights;
+this.rights = new ArrayList<>();
+this.rights.add(acl);
this.type = type;
-if (type == OzoneACLType.WORLD && name.length() != 0) {
+if (type == ACLIdentityType.WORLD && name.length() != 0) {
throw new IllegalArgumentException("Unexpected name part in world type");
}
-if (((type == OzoneACLType.USER) || (type == OzoneACLType.GROUP))
+if (((type == ACLIdentityType.USER) || (type == ACLIdentityType.GROUP))
+&& (name.length() == 0)) {
+ throw new IllegalArgumentException("User or group name is required");
+}
+ }
+
+ /**
+ * Constructor for OzoneAcl.
+ *
+ * @param type - Type
+ * @param name - Name of user
+ * @param acls - Rights
+ */
+ public OzoneAcl(ACLIdentityType type, String name, List acls) {
+this.name = name;
+this.rights = acls;
+this.type = type;
+if (type == ACLIdentityType.WORLD && name.length() != 0) {
Review comment:
Line 82-87 can be combined?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244198)
Time Spent: 50m (was: 40m)
> Update ozone protobuf message for ACLs
> --
>
> Key: HDDS-1538
> URL: https://issues.apache.org/jira/browse/HDDS-1538
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Update ozone protobuf message for ACLs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1538) Update ozone protobuf message for ACLs
[
https://issues.apache.org/jira/browse/HDDS-1538?focusedWorklogId=244196=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244196
]
ASF GitHub Bot logged work on HDDS-1538:
Author: ASF GitHub Bot
Created on: 17/May/19 18:49
Start Date: 17/May/19 18:49
Worklog Time Spent: 10m
Work Description: xiaoyuyao commented on pull request #828: HDDS-1538.
Update ozone protobuf message for ACLs. Contributed by Ajay Kumar.
URL: https://github.com/apache/hadoop/pull/828#discussion_r285247445
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OzoneAcl.java
##
@@ -47,16 +52,37 @@ public OzoneAcl() {
*
* @param type - Type
* @param name - Name of user
- * @param rights - Rights
+ * @param acl - Rights
*/
- public OzoneAcl(OzoneACLType type, String name, OzoneACLRights rights) {
+ public OzoneAcl(ACLIdentityType type, String name, ACLType acl) {
this.name = name;
-this.rights = rights;
+this.rights = new ArrayList<>();
+this.rights.add(acl);
this.type = type;
-if (type == OzoneACLType.WORLD && name.length() != 0) {
+if (type == ACLIdentityType.WORLD && name.length() != 0) {
throw new IllegalArgumentException("Unexpected name part in world type");
}
-if (((type == OzoneACLType.USER) || (type == OzoneACLType.GROUP))
+if (((type == ACLIdentityType.USER) || (type == ACLIdentityType.GROUP))
+&& (name.length() == 0)) {
+ throw new IllegalArgumentException("User or group name is required");
+}
+ }
+
+ /**
+ * Constructor for OzoneAcl.
+ *
+ * @param type - Type
+ * @param name - Name of user
Review comment:
This should be the name of the identity. Should we consider combine
(type,name) into a class?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244196)
Time Spent: 40m (was: 0.5h)
> Update ozone protobuf message for ACLs
> --
>
> Key: HDDS-1538
> URL: https://issues.apache.org/jira/browse/HDDS-1538
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Update ozone protobuf message for ACLs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1520) Ozone Classpath does not scale well using long path prefix to OZONE_HOME
[ https://issues.apache.org/jira/browse/HDDS-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842383#comment-16842383 ] Elek, Marton commented on HDDS-1520: bq. The solution is to make sure that we can put jar files in a directory, and use "lib/*" class path wildcards provided short hand to reference all jar files. I propose to use a different solution instead of the lib/* approach as it doesn't solve the problems which are discussed in HDDS-447. I think we can create helper jar files to define relative classpath in the Manifest (or just use the main artifacts with manifest entries). > Ozone Classpath does not scale well using long path prefix to OZONE_HOME > > > Key: HDDS-1520 > URL: https://issues.apache.org/jira/browse/HDDS-1520 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Elek, Marton >Priority: Major > > [~eyang] reported that the ozone can't be started if the ozone directory is > symlinked due to the classpath assembly. It should work even if the directory > is symlinked. > Ozone script generates CLASSPATH environment variable based on parsing of > ozone-0.5.0-SNAPSHOT/share/ozone/classpath/*.classpath file, then substitute > HDDS_LIB_JARS_DIR with OZONE_HOME/share/ozone/lib. When OZONE_HOME directory > path has a long prefix, this can cause CLASSPATH environment variable to > exceed string length limit set for a single environment variable. This cause > Ozone command to return random errors with class not found exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1538) Update ozone protobuf message for ACLs
[ https://issues.apache.org/jira/browse/HDDS-1538?focusedWorklogId=244194=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244194 ] ASF GitHub Bot logged work on HDDS-1538: Author: ASF GitHub Bot Created on: 17/May/19 18:43 Start Date: 17/May/19 18:43 Worklog Time Spent: 10m Work Description: xiaoyuyao commented on pull request #828: HDDS-1538. Update ozone protobuf message for ACLs. Contributed by Ajay Kumar. URL: https://github.com/apache/hadoop/pull/828#discussion_r285245362 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java ## @@ -56,8 +56,6 @@ public static final String OZONE_ACL_READ = "r"; Review comment: We will also need OZONE_ACL_CREATE, OZONE_ACL_READ_ACP, OZONE_ACL_WRITE_ACP as documented in the design spec. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244194) Time Spent: 0.5h (was: 20m) > Update ozone protobuf message for ACLs > -- > > Key: HDDS-1538 > URL: https://issues.apache.org/jira/browse/HDDS-1538 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Update ozone protobuf message for ACLs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1517) AllocateBlock call fails with ContainerNotFoundException
[ https://issues.apache.org/jira/browse/HDDS-1517?focusedWorklogId=244193=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244193 ] ASF GitHub Bot logged work on HDDS-1517: Author: ASF GitHub Bot Created on: 17/May/19 18:41 Start Date: 17/May/19 18:41 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #826: HDDS-1517. AllocateBlock call fails with ContainerNotFoundException. URL: https://github.com/apache/hadoop/pull/826#issuecomment-493557580 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 28 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 394 | trunk passed | | +1 | compile | 203 | trunk passed | | +1 | checkstyle | 50 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 815 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 125 | trunk passed | | 0 | spotbugs | 238 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 416 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 394 | the patch passed | | +1 | compile | 208 | the patch passed | | +1 | javac | 208 | the patch passed | | -0 | checkstyle | 31 | hadoop-hdds: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 662 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 127 | the patch passed | | +1 | findbugs | 423 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 131 | hadoop-hdds in the patch failed. | | -1 | unit | 955 | hadoop-ozone in the patch failed. | | +1 | asflicense | 37 | The patch does not generate ASF License warnings. | | | | 5170 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdds.scm.pipeline.TestRatisPipelineProvider | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-826/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/826 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 13d5d699ed50 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 4cb3da6 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-826/2/artifact/out/diff-checkstyle-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-826/2/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-826/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-826/2/testReport/ | | Max. process+thread count | 4557 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-826/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244193) Time Spent: 40m (was: 0.5h) > AllocateBlock call fails with ContainerNotFoundException > > > Key: HDDS-1517 > URL: https://issues.apache.org/jira/browse/HDDS-1517 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.5.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 >
[jira] [Commented] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842380#comment-16842380
]
Elek, Marton commented on HDDS-1458:
bq. I will make the new tests in the tarball when tests are written
I prefer to do it one step. Better to add something when it can be tested how
does it work. If it's not possible now, because there are now tests yet, let's
modify only the blockade part without the disk tests. I assume there will be a
new jira which explains the plan with the disk based fault-injection tests.
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Attachments: HDDS-1458.001.patch, HDDS-1458.002.patch,
> HDDS-1458.003.patch, HDDS-1458.004.patch, HDDS-1458.005.patch,
> HDDS-1458.006.patch, HDDS-1458.007.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14323) Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path
[
https://issues.apache.org/jira/browse/HDFS-14323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842345#comment-16842345
]
Srinivasu Majeti commented on HDFS-14323:
-
Thank you very much [~jojochuang] for your time to review and approval.
> Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters
> in hdfs file path
> ---
>
> Key: HDFS-14323
> URL: https://issues.apache.org/jira/browse/HDFS-14323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.2.0
>Reporter: Srinivasu Majeti
>Assignee: Srinivasu Majeti
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14323v0.patch
>
>
> There was an enhancement to allow semicolon in source/target URLs for distcp
> use case as part of HDFS-13176 and backward compatibility fix as part of
> HDFS-13582 . Still there seems to be an issue when trying to trigger distcp
> from 3.x cluster to pull webhdfs data from 2.x hadoop cluster. We might need
> to deal with existing fix as described below by making sure if url is already
> encoded or not. That fixes it.
> diff --git
> a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
>
> b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> index 5936603c34a..dc790286aff 100644
> ---
> a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> +++
> b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> @@ -609,7 +609,10 @@ URL toUrl(final HttpOpParam.Op op, final Path fspath,
> boolean pathAlreadyEncoded = false;
> try {
> fspathUriDecoded = URLDecoder.decode(fspathUri.getPath(), "UTF-8");
> - pathAlreadyEncoded = true;
> + if(!fspathUri.getPath().equals(fspathUriDecoded))
> + {
> + pathAlreadyEncoded = true;
> + }
> } catch (IllegalArgumentException ex) {
> LOG.trace("Cannot decode URL encoded file", ex);
> }
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14499) Misleading REM_QUOTA value with snasphot and trash feature enabled for a directory
[
https://issues.apache.org/jira/browse/HDFS-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Chitlangia reassigned HDFS-14499:
Assignee: (was: Dinesh Chitlangia)
> Misleading REM_QUOTA value with snasphot and trash feature enabled for a
> directory
> --
>
> Key: HDFS-14499
> URL: https://issues.apache.org/jira/browse/HDFS-14499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Shashikant Banerjee
>Priority: Major
>
> This is the flow of steps where we see a discrepancy between REM_QUOTA and
> new file operation failure. REM_QUOTA shows a value of 1 but file creation
> operation does not succeed.
> {code:java}
> hdfs@c3265-node3 root$ hdfs dfs -mkdir /dir1
> hdfs@c3265-node3 root$ hdfs dfsadmin -setQuota 2 /dir1
> hdfs@c3265-node3 root$ hdfs dfsadmin -allowSnapshot /dir1
> Allowing snaphot on /dir1 succeeded
> hdfs@c3265-node3 root$ hdfs dfs -touchz /dir1/file1
> hdfs@c3265-node3 root$ hdfs dfs -createSnapshot /dir1 snap1
> Created snapshot /dir1/.snapshot/snap1
> hdfs@c3265-node3 root$ hdfs dfs -count -v -q /dir1
> QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE
> PATHNAME
> 2 0 none inf 1 1 0 /dir1
> hdfs@c3265-node3 root$ hdfs dfs -rm /dir1/file1
> 19/03/26 11:20:25 INFO fs.TrashPolicyDefault: Moved:
> 'hdfs://smajetinn/dir1/file1' to trash at:
> hdfs://smajetinn/user/hdfs/.Trash/Current/dir1/file11553599225772
> hdfs@c3265-node3 root$ hdfs dfs -count -v -q /dir1
> QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE
> PATHNAME
> 2 1 none inf 1 0 0 /dir1
> hdfs@c3265-node3 root$ hdfs dfs -touchz /dir1/file1
> touchz: The NameSpace quota (directories and files) of directory /dir1 is
> exceeded: quota=2 file count=3{code}
> The issue here, is that the count command takes only files and directories
> into account not the inode references. When trash is enabled, the deletion of
> files inside a directory actually does a rename operation as a result of
> which an inode reference is maintained in the deleted list of the snapshot
> diff which is taken into account while computing the namespace quota, but
> count command (getContentSummary()) ,just takes into account just the files
> and directories, not the referenced entity for calculating the REM_QUOTA. The
> referenced entity is taken into account for space quota only.
> InodeReference.java:
> ---
> {code:java}
> @Override
> public final ContentSummaryComputationContext computeContentSummary(
> int snapshotId, ContentSummaryComputationContext summary) {
> final int s = snapshotId < lastSnapshotId ? snapshotId : lastSnapshotId;
> // only count storagespace for WithName
> final QuotaCounts q = computeQuotaUsage(
> summary.getBlockStoragePolicySuite(), getStoragePolicyID(), false,
> s);
> summary.getCounts().addContent(Content.DISKSPACE, q.getStorageSpace());
> summary.getCounts().addTypeSpaces(q.getTypeSpaces());
> return summary;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14323) Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path
[
https://issues.apache.org/jira/browse/HDFS-14323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842340#comment-16842340
]
Hudson commented on HDFS-14323:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16568 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16568/])
HDFS-14323. Distcp fails in Hadoop 3.x when 2.x source webhdfs url has
(weichiu: rev 3e5e5b028ad7e199d08e524fe7cddeee5db51a6d)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters
> in hdfs file path
> ---
>
> Key: HDFS-14323
> URL: https://issues.apache.org/jira/browse/HDFS-14323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.2.0
>Reporter: Srinivasu Majeti
>Assignee: Srinivasu Majeti
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14323v0.patch
>
>
> There was an enhancement to allow semicolon in source/target URLs for distcp
> use case as part of HDFS-13176 and backward compatibility fix as part of
> HDFS-13582 . Still there seems to be an issue when trying to trigger distcp
> from 3.x cluster to pull webhdfs data from 2.x hadoop cluster. We might need
> to deal with existing fix as described below by making sure if url is already
> encoded or not. That fixes it.
> diff --git
> a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
>
> b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> index 5936603c34a..dc790286aff 100644
> ---
> a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> +++
> b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> @@ -609,7 +609,10 @@ URL toUrl(final HttpOpParam.Op op, final Path fspath,
> boolean pathAlreadyEncoded = false;
> try {
> fspathUriDecoded = URLDecoder.decode(fspathUri.getPath(), "UTF-8");
> - pathAlreadyEncoded = true;
> + if(!fspathUri.getPath().equals(fspathUriDecoded))
> + {
> + pathAlreadyEncoded = true;
> + }
> } catch (IllegalArgumentException ex) {
> LOG.trace("Cannot decode URL encoded file", ex);
> }
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14498) LeaseManager can loop forever on the file for which create has failed
[
https://issues.apache.org/jira/browse/HDFS-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842339#comment-16842339
]
Sergey Shelukhin commented on HDFS-14498:
-
I don't have a repro because this issue also basically destroys namenode logs.
I'm assuming judging by the fact that the file blocks cannot be replicated that
write failed at some stage and/or client died. The file only has one block, and
that block is the one without replicas.
> LeaseManager can loop forever on the file for which create has failed
> --
>
> Key: HDFS-14498
> URL: https://issues.apache.org/jira/browse/HDFS-14498
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Sergey Shelukhin
>Priority: Major
>
> The logs from file creation are long gone due to infinite lease logging,
> however it presumably failed... the client who was trying to write this file
> is definitely long dead.
> The version includes HDFS-4882.
> We get this log pattern repeating infinitely:
> {noformat}
> 2019-05-16 14:00:16,893 INFO
> [org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@b27557f]
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: [Lease. Holder:
> DFSClient_NONMAPREDUCE_-20898906_61, pending creates: 1] has expired hard
> limit
> 2019-05-16 14:00:16,893 INFO
> [org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@b27557f]
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering [Lease.
> Holder: DFSClient_NONMAPREDUCE_-20898906_61, pending creates: 1], src=
> 2019-05-16 14:00:16,893 WARN
> [org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@b27557f]
> org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.internalReleaseLease:
> Failed to release lease for file . Committed blocks are waiting to be
> minimally replicated. Try again later.
> 2019-05-16 14:00:16,893 WARN
> [org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@b27557f]
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: Cannot release the path
> in the lease [Lease. Holder: DFSClient_NONMAPREDUCE_-20898906_61,
> pending creates: 1]. It will be retried.
> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: DIR*
> NameSystem.internalReleaseLease: Failed to release lease for file .
> Committed blocks are waiting to be minimally replicated. Try again later.
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.internalReleaseLease(FSNamesystem.java:3357)
> at
> org.apache.hadoop.hdfs.server.namenode.LeaseManager.checkLeases(LeaseManager.java:573)
> at
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:509)
> at java.lang.Thread.run(Thread.java:745)
> $ grep -c "Recovering.*DFSClient_NONMAPREDUCE_-20898906_61, pending creates:
> 1" hdfs_nn*
> hdfs_nn.log:1068035
> hdfs_nn.log.2019-05-16-14:1516179
> hdfs_nn.log.2019-05-16-15:1538350
> {noformat}
> Aside from an actual bug fix, it might make sense to make LeaseManager not
> log so much, in case if there are more bugs like this...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14323) Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path
[
https://issues.apache.org/jira/browse/HDFS-14323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842327#comment-16842327
]
Wei-Chiu Chuang edited comment on HDFS-14323 at 5/17/19 5:30 PM:
-
+1
I stared at the patch for several hours and finally understood it. Thanks. This
is good stuff.
(I used https://www.urldecoder.org/ and https://www.urlencoder.org/, and
entered the path /tmp/date=1234557/check to verify the result of URL
decode/encode)
bq. The current issue is completely to address HDP 3.1 to HDP 2.6 webhdfs and
which cannot be Junit tested as we cant trigger two clusters (one for each
version )
I now realized the test added in HDFS-13582 passes regardless of the fix of
HDFS-13582.
was (Author: jojochuang):
+1
I stared at the patch for several hours and finally understood it. Thanks. This
is good stuff.
(I used https://www.urldecoder.org/ and https://www.urlencoder.org/, and
entered the path /tmp/date=1234557/check for test)
bq. The current issue is completely to address HDP 3.1 to HDP 2.6 webhdfs and
which cannot be Junit tested as we cant trigger two clusters (one for each
version )
I now realized the test added in HDFS-13582 passes regardless of the fix of
HDFS-13582.
> Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters
> in hdfs file path
> ---
>
> Key: HDFS-14323
> URL: https://issues.apache.org/jira/browse/HDFS-14323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.2.0
>Reporter: Srinivasu Majeti
>Assignee: Srinivasu Majeti
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14323v0.patch
>
>
> There was an enhancement to allow semicolon in source/target URLs for distcp
> use case as part of HDFS-13176 and backward compatibility fix as part of
> HDFS-13582 . Still there seems to be an issue when trying to trigger distcp
> from 3.x cluster to pull webhdfs data from 2.x hadoop cluster. We might need
> to deal with existing fix as described below by making sure if url is already
> encoded or not. That fixes it.
> diff --git
> a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
>
> b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> index 5936603c34a..dc790286aff 100644
> ---
> a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> +++
> b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> @@ -609,7 +609,10 @@ URL toUrl(final HttpOpParam.Op op, final Path fspath,
> boolean pathAlreadyEncoded = false;
> try {
> fspathUriDecoded = URLDecoder.decode(fspathUri.getPath(), "UTF-8");
> - pathAlreadyEncoded = true;
> + if(!fspathUri.getPath().equals(fspathUriDecoded))
> + {
> + pathAlreadyEncoded = true;
> + }
> } catch (IllegalArgumentException ex) {
> LOG.trace("Cannot decode URL encoded file", ex);
> }
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14323) Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path
[
https://issues.apache.org/jira/browse/HDFS-14323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14323:
---
Fix Version/s: 3.1.3
3.2.1
3.3.0
> Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters
> in hdfs file path
> ---
>
> Key: HDFS-14323
> URL: https://issues.apache.org/jira/browse/HDFS-14323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.2.0
>Reporter: Srinivasu Majeti
>Assignee: Srinivasu Majeti
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14323v0.patch
>
>
> There was an enhancement to allow semicolon in source/target URLs for distcp
> use case as part of HDFS-13176 and backward compatibility fix as part of
> HDFS-13582 . Still there seems to be an issue when trying to trigger distcp
> from 3.x cluster to pull webhdfs data from 2.x hadoop cluster. We might need
> to deal with existing fix as described below by making sure if url is already
> encoded or not. That fixes it.
> diff --git
> a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
>
> b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> index 5936603c34a..dc790286aff 100644
> ---
> a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> +++
> b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> @@ -609,7 +609,10 @@ URL toUrl(final HttpOpParam.Op op, final Path fspath,
> boolean pathAlreadyEncoded = false;
> try {
> fspathUriDecoded = URLDecoder.decode(fspathUri.getPath(), "UTF-8");
> - pathAlreadyEncoded = true;
> + if(!fspathUri.getPath().equals(fspathUriDecoded))
> + {
> + pathAlreadyEncoded = true;
> + }
> } catch (IllegalArgumentException ex) {
> LOG.trace("Cannot decode URL encoded file", ex);
> }
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14323) Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path
[
https://issues.apache.org/jira/browse/HDFS-14323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14323:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters
> in hdfs file path
> ---
>
> Key: HDFS-14323
> URL: https://issues.apache.org/jira/browse/HDFS-14323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.2.0
>Reporter: Srinivasu Majeti
>Assignee: Srinivasu Majeti
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14323v0.patch
>
>
> There was an enhancement to allow semicolon in source/target URLs for distcp
> use case as part of HDFS-13176 and backward compatibility fix as part of
> HDFS-13582 . Still there seems to be an issue when trying to trigger distcp
> from 3.x cluster to pull webhdfs data from 2.x hadoop cluster. We might need
> to deal with existing fix as described below by making sure if url is already
> encoded or not. That fixes it.
> diff --git
> a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
>
> b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> index 5936603c34a..dc790286aff 100644
> ---
> a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> +++
> b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> @@ -609,7 +609,10 @@ URL toUrl(final HttpOpParam.Op op, final Path fspath,
> boolean pathAlreadyEncoded = false;
> try {
> fspathUriDecoded = URLDecoder.decode(fspathUri.getPath(), "UTF-8");
> - pathAlreadyEncoded = true;
> + if(!fspathUri.getPath().equals(fspathUriDecoded))
> + {
> + pathAlreadyEncoded = true;
> + }
> } catch (IllegalArgumentException ex) {
> LOG.trace("Cannot decode URL encoded file", ex);
> }
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13582) Improve backward compatibility for HDFS-13176 (WebHdfs file path gets truncated when having semicolon (;) inside)
[ https://issues.apache.org/jira/browse/HDFS-13582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842333#comment-16842333 ] Wei-Chiu Chuang commented on HDFS-13582: Added 3.1.1 to fixed version. > Improve backward compatibility for HDFS-13176 (WebHdfs file path gets > truncated when having semicolon (;) inside) > - > > Key: HDFS-13582 > URL: https://issues.apache.org/jira/browse/HDFS-13582 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Zsolt Venczel >Assignee: Zsolt Venczel >Priority: Major > Fix For: 3.2.0, 3.1.1 > > Attachments: HDFS-13582-branch-2.01.patch, HDFS-13582.01.patch, > HDFS-13582.02.patch > > > Encode special character only if necessary in order to improve backward > compatibility in the following scenario: > new (having HDFS-13176) WebHdfs client - > old (not having HDFS-13176) > WebHdfs server -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13582) Improve backward compatibility for HDFS-13176 (WebHdfs file path gets truncated when having semicolon (;) inside)
[ https://issues.apache.org/jira/browse/HDFS-13582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-13582: --- Fix Version/s: 3.1.1 > Improve backward compatibility for HDFS-13176 (WebHdfs file path gets > truncated when having semicolon (;) inside) > - > > Key: HDFS-13582 > URL: https://issues.apache.org/jira/browse/HDFS-13582 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Zsolt Venczel >Assignee: Zsolt Venczel >Priority: Major > Fix For: 3.2.0, 3.1.1 > > Attachments: HDFS-13582-branch-2.01.patch, HDFS-13582.01.patch, > HDFS-13582.02.patch > > > Encode special character only if necessary in order to improve backward > compatibility in the following scenario: > new (having HDFS-13176) WebHdfs client - > old (not having HDFS-13176) > WebHdfs server -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14323) Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path
[
https://issues.apache.org/jira/browse/HDFS-14323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842327#comment-16842327
]
Wei-Chiu Chuang commented on HDFS-14323:
+1
I stared at the patch for several hours and finally understood it. Thanks. This
is good stuff.
(I used https://www.urldecoder.org/ and https://www.urlencoder.org/, and
entered the path /tmp/date=1234557/check for test)
bq. The current issue is completely to address HDP 3.1 to HDP 2.6 webhdfs and
which cannot be Junit tested as we cant trigger two clusters (one for each
version )
I now realized the test added in HDFS-13582 passes regardless of the fix of
HDFS-13582.
> Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters
> in hdfs file path
> ---
>
> Key: HDFS-14323
> URL: https://issues.apache.org/jira/browse/HDFS-14323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.2.0
>Reporter: Srinivasu Majeti
>Assignee: Srinivasu Majeti
>Priority: Major
> Attachments: HDFS-14323v0.patch
>
>
> There was an enhancement to allow semicolon in source/target URLs for distcp
> use case as part of HDFS-13176 and backward compatibility fix as part of
> HDFS-13582 . Still there seems to be an issue when trying to trigger distcp
> from 3.x cluster to pull webhdfs data from 2.x hadoop cluster. We might need
> to deal with existing fix as described below by making sure if url is already
> encoded or not. That fixes it.
> diff --git
> a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
>
> b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> index 5936603c34a..dc790286aff 100644
> ---
> a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> +++
> b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> @@ -609,7 +609,10 @@ URL toUrl(final HttpOpParam.Op op, final Path fspath,
> boolean pathAlreadyEncoded = false;
> try {
> fspathUriDecoded = URLDecoder.decode(fspathUri.getPath(), "UTF-8");
> - pathAlreadyEncoded = true;
> + if(!fspathUri.getPath().equals(fspathUriDecoded))
> + {
> + pathAlreadyEncoded = true;
> + }
> } catch (IllegalArgumentException ex) {
> LOG.trace("Cannot decode URL encoded file", ex);
> }
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1517) AllocateBlock call fails with ContainerNotFoundException
[ https://issues.apache.org/jira/browse/HDDS-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842325#comment-16842325 ] Shashikant Banerjee commented on HDDS-1517: --- Thanks [~jnp], as discussed i have updated the patch in the pull request. > AllocateBlock call fails with ContainerNotFoundException > > > Key: HDDS-1517 > URL: https://issues.apache.org/jira/browse/HDDS-1517 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.5.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Attachments: HDDS-1517.000.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > In allocateContainer call, the container is first added to pipelineStateMap > and then added to container cache. If two allocate blocks execute > concurrently, it might happen that one find the container to exist in the > pipelineStateMap but the container is yet to be updated in the container > cache, hence failing with CONTAINER_NOT_FOUND exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14090) RBF: Improved isolation for downstream name nodes.
[
https://issues.apache.org/jira/browse/HDFS-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842316#comment-16842316
]
CR Hota commented on HDFS-14090:
[~elgoiri] [~ajisakaa] Thanks for the review.
{quote}
In the past few months I've been doing some changes to support unavailable
subclusters.
I think this approach would be able to handle this a little more cleanly.
It would be nice to have a summary of where the main code changes would be.
It looks to me that we will have to do some change to the commons part for both
RPC client and server.{quote}
Essentially in RouterRpcClient after a lookup is done and invoke is about to be
called, permit will be requested. During init of routers, RouterRpcClient will
hold a reference to FairnessManager that is the permit granting object. Will
put a patch soon.
{quote}It would be very nice if DFSRouter is available via service discovery.
(HADOOP-15774) {quote}
Routers can be configured behind a DNS. HDFS-14118
> RBF: Improved isolation for downstream name nodes.
> --
>
> Key: HDFS-14090
> URL: https://issues.apache.org/jira/browse/HDFS-14090
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Attachments: RBF_ Isolation design.pdf
>
>
> Router is a gateway to underlying name nodes. Gateway architectures, should
> help minimize impact of clients connecting to healthy clusters vs unhealthy
> clusters.
> For example - If there are 2 name nodes downstream, and one of them is
> heavily loaded with calls spiking rpc queue times, due to back pressure the
> same with start reflecting on the router. As a result of this, clients
> connecting to healthy/faster name nodes will also slow down as same rpc queue
> is maintained for all calls at the router layer. Essentially the same IPC
> thread pool is used by router to connect to all name nodes.
> Currently router uses one single rpc queue for all calls. Lets discuss how we
> can change the architecture and add some throttling logic for
> unhealthy/slow/overloaded name nodes.
> One way could be to read from current call queue, immediately identify
> downstream name node and maintain a separate queue for each underlying name
> node. Another simpler way is to maintain some sort of rate limiter configured
> for each name node and let routers drop/reject/send error requests after
> certain threshold.
> This won’t be a simple change as router’s ‘Server’ layer would need redesign
> and implementation. Currently this layer is the same as name node.
> Opening this ticket to discuss, design and implement this feature.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14186) blockreport storm slow down namenode restart seriously in large cluster
[ https://issues.apache.org/jira/browse/HDFS-14186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842303#comment-16842303 ] Kihwal Lee commented on HDFS-14186: --- We probably want an ability to tell whether a block report has been received for each and every known storage. If the safe block count is reached AND the nn received all block reports that are supposed to come, it will do only the necessary amount of replication when coming out of safe mode. This is different from waiting for all replica locations for all blocks, which will not happen if any node is down. A similar concept is there as "staleness" check after a HA transition. Standby safe mode can optionally get the list of nodes from the active, if running, and do this check to determine whether it is truly ready to take over. > blockreport storm slow down namenode restart seriously in large cluster > --- > > Key: HDFS-14186 > URL: https://issues.apache.org/jira/browse/HDFS-14186 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.1 >Reporter: He Xiaoqiao >Assignee: He Xiaoqiao >Priority: Major > Attachments: HDFS-14186.001.patch > > > In the current implementation, the datanode sends blockreport immediately > after register to namenode successfully when restart, and the blockreport > storm will make namenode high load to process them. One result is some > received RPC have to skip because queue time is timeout. If some datanodes' > heartbeat RPC are continually skipped for long times (default is > heartbeatExpireInterval=630s) it will be set DEAD, then datanode has to > re-register and send blockreport again, aggravate blockreport storm and trap > in a vicious circle, and slow down (more than one hour and even more) > namenode startup seriously in a large (several thousands of datanodes) and > busy cluster especially. Although there are many work to optimize namenode > startup, the issue still exists. > I propose to postpone dead datanode check when namenode have finished startup. > Any comments and suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1550) MiniOzoneCluster is not shutting down all the threads during shutdown.
[
https://issues.apache.org/jira/browse/HDDS-1550?focusedWorklogId=244120=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244120
]
ASF GitHub Bot logged work on HDDS-1550:
Author: ASF GitHub Bot
Created on: 17/May/19 16:13
Start Date: 17/May/19 16:13
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #829:
HDDS-1550. MiniOzoneCluster is not shutting down all the threads during
shutdown. Contributed by Mukul Kumar Singh.
URL: https://github.com/apache/hadoop/pull/829#discussion_r285193337
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerGrpc.java
##
@@ -172,6 +173,11 @@ public void start() throws IOException {
public void stop() {
if (isStarted) {
server.shutdown();
+ try {
+server.awaitTermination(1, TimeUnit.DAYS);
Review comment:
Same question as above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244120)
Time Spent: 1.5h (was: 1h 20m)
> MiniOzoneCluster is not shutting down all the threads during shutdown.
> --
>
> Key: HDDS-1550
> URL: https://issues.apache.org/jira/browse/HDDS-1550
> Project: Hadoop Distributed Data Store
> Issue Type: Test
> Components: test
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> MiniOzoneCluster does not shutdown all the threads during shutdown. All the
> threads must be shutdown to close the cluster correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1550) MiniOzoneCluster is not shutting down all the threads during shutdown.
[
https://issues.apache.org/jira/browse/HDDS-1550?focusedWorklogId=244119=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244119
]
ASF GitHub Bot logged work on HDDS-1550:
Author: ASF GitHub Bot
Created on: 17/May/19 16:12
Start Date: 17/May/19 16:12
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #829:
HDDS-1550. MiniOzoneCluster is not shutting down all the threads during
shutdown. Contributed by Mukul Kumar Singh.
URL: https://github.com/apache/hadoop/pull/829#discussion_r285193246
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/report/ReportManager.java
##
@@ -71,6 +76,11 @@ public void init() {
*/
public void shutdown() {
executorService.shutdown();
+try {
+ executorService.awaitTermination(1, TimeUnit.DAYS);
+} catch (Exception e) {
+ LOG.error("failed to shutdown Report Manager", e);
Review comment:
Do you want timeUnit to be Days?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244119)
Time Spent: 1h 20m (was: 1h 10m)
> MiniOzoneCluster is not shutting down all the threads during shutdown.
> --
>
> Key: HDDS-1550
> URL: https://issues.apache.org/jira/browse/HDDS-1550
> Project: Hadoop Distributed Data Store
> Issue Type: Test
> Components: test
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> MiniOzoneCluster does not shutdown all the threads during shutdown. All the
> threads must be shutdown to close the cluster correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1550) MiniOzoneCluster is not shutting down all the threads during shutdown.
[ https://issues.apache.org/jira/browse/HDDS-1550?focusedWorklogId=244106=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244106 ] ASF GitHub Bot logged work on HDDS-1550: Author: ASF GitHub Bot Created on: 17/May/19 15:53 Start Date: 17/May/19 15:53 Worklog Time Spent: 10m Work Description: bshashikant commented on issue #829: HDDS-1550. MiniOzoneCluster is not shutting down all the threads during shutdown. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop/pull/829#issuecomment-493504166 Thanks @mukul1987 for working on this. i am +1 on this. The patch looks good to me. Please address @jiwq comments, while committing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244106) Time Spent: 1h 10m (was: 1h) > MiniOzoneCluster is not shutting down all the threads during shutdown. > -- > > Key: HDDS-1550 > URL: https://issues.apache.org/jira/browse/HDDS-1550 > Project: Hadoop Distributed Data Store > Issue Type: Test > Components: test >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > MiniOzoneCluster does not shutdown all the threads during shutdown. All the > threads must be shutdown to close the cluster correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1550) MiniOzoneCluster is not shutting down all the threads during shutdown.
[ https://issues.apache.org/jira/browse/HDDS-1550?focusedWorklogId=244105=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244105 ] ASF GitHub Bot logged work on HDDS-1550: Author: ASF GitHub Bot Created on: 17/May/19 15:52 Start Date: 17/May/19 15:52 Worklog Time Spent: 10m Work Description: bshashikant commented on issue #829: HDDS-1550. MiniOzoneCluster is not shutting down all the threads during shutdown. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop/pull/829#issuecomment-493504166 Thanks @mukul1987 for working on this. The patch looks good to me. Please address @jiwq comments, while committing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244105) Time Spent: 1h (was: 50m) > MiniOzoneCluster is not shutting down all the threads during shutdown. > -- > > Key: HDDS-1550 > URL: https://issues.apache.org/jira/browse/HDDS-1550 > Project: Hadoop Distributed Data Store > Issue Type: Test > Components: test >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > MiniOzoneCluster does not shutdown all the threads during shutdown. All the > threads must be shutdown to close the cluster correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13255) RBF: Fail when try to remove mount point paths
[
https://issues.apache.org/jira/browse/HDFS-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842268#comment-16842268
]
Ayush Saxena commented on HDFS-13255:
-
As of now if perform any operation like SetOwner(..) on a mount Point, it sets
the owner on the destination and works fine as expected.
Does restricting this behavior has any advantage?
> RBF: Fail when try to remove mount point paths
> --
>
> Key: HDFS-13255
> URL: https://issues.apache.org/jira/browse/HDFS-13255
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-13255-HDFS-13891-wip-001.patch
>
>
> when delete a ns-fed path which include mount point paths, will issue a error.
> Need to delete each mount point path independently.
> Operation step:
> {code:java}
> [hadp@root]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> Source Destinations Owner Group Mode Quota/Usage
> /rm-test-all/rm-test-ns10 ns10->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> /rm-test-all/rm-test-ns2 ns1->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns2/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 101 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/NOTICE.txt
> -rw-r--r-- 3 hadp supergroup 1366 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/README.txt
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -rm -r hdfs://ns-fed/rm-test-all/
> rm: Failed to move to trash: hdfs://ns-fed/rm-test-all. Consider using
> -skipTrash option
> [hadp@root]$ hdfs dfs -rm -r -skipTrash hdfs://ns-fed/rm-test-all/
> rm: `hdfs://ns-fed/rm-test-all': Input/output error
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1550) MiniOzoneCluster is not shutting down all the threads during shutdown.
[ https://issues.apache.org/jira/browse/HDDS-1550?focusedWorklogId=244081=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244081 ] ASF GitHub Bot logged work on HDDS-1550: Author: ASF GitHub Bot Created on: 17/May/19 14:43 Start Date: 17/May/19 14:43 Worklog Time Spent: 10m Work Description: jiwq commented on pull request #829: HDDS-1550. MiniOzoneCluster is not shutting down all the threads during shutdown. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop/pull/829#discussion_r285151145 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java ## @@ -205,6 +206,7 @@ private final OzoneConfiguration configuration; private final SafeModeHandler safeModeHandler; private SCMContainerMetrics scmContainerMetrics; + private MetricsSystem ms; Review comment: Suggest modify the instance variable name, so that it readability. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244081) Time Spent: 40m (was: 0.5h) > MiniOzoneCluster is not shutting down all the threads during shutdown. > -- > > Key: HDDS-1550 > URL: https://issues.apache.org/jira/browse/HDDS-1550 > Project: Hadoop Distributed Data Store > Issue Type: Test > Components: test >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > MiniOzoneCluster does not shutdown all the threads during shutdown. All the > threads must be shutdown to close the cluster correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1550) MiniOzoneCluster is not shutting down all the threads during shutdown.
[
https://issues.apache.org/jira/browse/HDDS-1550?focusedWorklogId=244082=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244082
]
ASF GitHub Bot logged work on HDDS-1550:
Author: ASF GitHub Bot
Created on: 17/May/19 14:43
Start Date: 17/May/19 14:43
Worklog Time Spent: 10m
Work Description: jiwq commented on pull request #829: HDDS-1550.
MiniOzoneCluster is not shutting down all the threads during shutdown.
Contributed by Mukul Kumar Singh.
URL: https://github.com/apache/hadoop/pull/829#discussion_r285148899
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/report/ReportManager.java
##
@@ -71,6 +76,11 @@ public void init() {
*/
public void shutdown() {
executorService.shutdown();
+try {
+ executorService.awaitTermination(1, TimeUnit.DAYS);
+} catch (Exception e) {
+ LOG.error("failed to shutdown Report Manager", e);
Review comment:
The first letter of the first word as a capital letter maybe better.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244082)
Time Spent: 50m (was: 40m)
> MiniOzoneCluster is not shutting down all the threads during shutdown.
> --
>
> Key: HDDS-1550
> URL: https://issues.apache.org/jira/browse/HDDS-1550
> Project: Hadoop Distributed Data Store
> Issue Type: Test
> Components: test
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> MiniOzoneCluster does not shutdown all the threads during shutdown. All the
> threads must be shutdown to close the cluster correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1550) MiniOzoneCluster is not shutting down all the threads during shutdown.
[
https://issues.apache.org/jira/browse/HDDS-1550?focusedWorklogId=244080=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244080
]
ASF GitHub Bot logged work on HDDS-1550:
Author: ASF GitHub Bot
Created on: 17/May/19 14:43
Start Date: 17/May/19 14:43
Worklog Time Spent: 10m
Work Description: jiwq commented on pull request #829: HDDS-1550.
MiniOzoneCluster is not shutting down all the threads during shutdown.
Contributed by Mukul Kumar Singh.
URL: https://github.com/apache/hadoop/pull/829#discussion_r285152336
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneChaosCluster.java
##
@@ -203,7 +203,7 @@ void initializeConfiguration() throws IOException {
1, TimeUnit.SECONDS);
conf.setTimeDuration(HddsConfigKeys.HDDS_HEARTBEAT_INTERVAL, 1,
TimeUnit.SECONDS);
- conf.setInt(OzoneConfigKeys.OZONE_CONTAINER_CACHE_SIZE, 8);
+ conf.setInt(OzoneConfigKeys.OZONE_CONTAINER_CACHE_SIZE, 2);
Review comment:
What's the purpose of change the cache size?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244080)
Time Spent: 0.5h (was: 20m)
> MiniOzoneCluster is not shutting down all the threads during shutdown.
> --
>
> Key: HDDS-1550
> URL: https://issues.apache.org/jira/browse/HDDS-1550
> Project: Hadoop Distributed Data Store
> Issue Type: Test
> Components: test
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> MiniOzoneCluster does not shutdown all the threads during shutdown. All the
> threads must be shutdown to close the cluster correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1550) MiniOzoneCluster is not shutting down all the threads during shutdown.
[
https://issues.apache.org/jira/browse/HDDS-1550?focusedWorklogId=244083=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244083
]
ASF GitHub Bot logged work on HDDS-1550:
Author: ASF GitHub Bot
Created on: 17/May/19 14:43
Start Date: 17/May/19 14:43
Worklog Time Spent: 10m
Work Description: jiwq commented on pull request #829: HDDS-1550.
MiniOzoneCluster is not shutting down all the threads during shutdown.
Contributed by Mukul Kumar Singh.
URL: https://github.com/apache/hadoop/pull/829#discussion_r285156634
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerGrpc.java
##
@@ -172,6 +173,11 @@ public void start() throws IOException {
public void stop() {
if (isStarted) {
server.shutdown();
+ try {
+server.awaitTermination(1, TimeUnit.DAYS);
Review comment:
It's better to read the value from configuration file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244083)
> MiniOzoneCluster is not shutting down all the threads during shutdown.
> --
>
> Key: HDDS-1550
> URL: https://issues.apache.org/jira/browse/HDDS-1550
> Project: Hadoop Distributed Data Store
> Issue Type: Test
> Components: test
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> MiniOzoneCluster does not shutdown all the threads during shutdown. All the
> threads must be shutdown to close the cluster correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14499) Misleading REM_QUOTA value with snasphot and trash feature enabled for a directory
[
https://issues.apache.org/jira/browse/HDFS-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Chitlangia reassigned HDFS-14499:
Assignee: Dinesh Chitlangia
> Misleading REM_QUOTA value with snasphot and trash feature enabled for a
> directory
> --
>
> Key: HDFS-14499
> URL: https://issues.apache.org/jira/browse/HDFS-14499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Shashikant Banerjee
>Assignee: Dinesh Chitlangia
>Priority: Major
>
> This is the flow of steps where we see a discrepancy between REM_QUOTA and
> new file operation failure. REM_QUOTA shows a value of 1 but file creation
> operation does not succeed.
> {code:java}
> hdfs@c3265-node3 root$ hdfs dfs -mkdir /dir1
> hdfs@c3265-node3 root$ hdfs dfsadmin -setQuota 2 /dir1
> hdfs@c3265-node3 root$ hdfs dfsadmin -allowSnapshot /dir1
> Allowing snaphot on /dir1 succeeded
> hdfs@c3265-node3 root$ hdfs dfs -touchz /dir1/file1
> hdfs@c3265-node3 root$ hdfs dfs -createSnapshot /dir1 snap1
> Created snapshot /dir1/.snapshot/snap1
> hdfs@c3265-node3 root$ hdfs dfs -count -v -q /dir1
> QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE
> PATHNAME
> 2 0 none inf 1 1 0 /dir1
> hdfs@c3265-node3 root$ hdfs dfs -rm /dir1/file1
> 19/03/26 11:20:25 INFO fs.TrashPolicyDefault: Moved:
> 'hdfs://smajetinn/dir1/file1' to trash at:
> hdfs://smajetinn/user/hdfs/.Trash/Current/dir1/file11553599225772
> hdfs@c3265-node3 root$ hdfs dfs -count -v -q /dir1
> QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE
> PATHNAME
> 2 1 none inf 1 0 0 /dir1
> hdfs@c3265-node3 root$ hdfs dfs -touchz /dir1/file1
> touchz: The NameSpace quota (directories and files) of directory /dir1 is
> exceeded: quota=2 file count=3{code}
> The issue here, is that the count command takes only files and directories
> into account not the inode references. When trash is enabled, the deletion of
> files inside a directory actually does a rename operation as a result of
> which an inode reference is maintained in the deleted list of the snapshot
> diff which is taken into account while computing the namespace quota, but
> count command (getContentSummary()) ,just takes into account just the files
> and directories, not the referenced entity for calculating the REM_QUOTA. The
> referenced entity is taken into account for space quota only.
> InodeReference.java:
> ---
> {code:java}
> @Override
> public final ContentSummaryComputationContext computeContentSummary(
> int snapshotId, ContentSummaryComputationContext summary) {
> final int s = snapshotId < lastSnapshotId ? snapshotId : lastSnapshotId;
> // only count storagespace for WithName
> final QuotaCounts q = computeQuotaUsage(
> summary.getBlockStoragePolicySuite(), getStoragePolicyID(), false,
> s);
> summary.getCounts().addContent(Content.DISKSPACE, q.getStorageSpace());
> summary.getCounts().addTypeSpaces(q.getTypeSpaces());
> return summary;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14499) Misleading REM_QUOTA value with snasphot and trash feature enabled for a directory
Shashikant Banerjee created HDFS-14499:
--
Summary: Misleading REM_QUOTA value with snasphot and trash
feature enabled for a directory
Key: HDFS-14499
URL: https://issues.apache.org/jira/browse/HDFS-14499
Project: Hadoop HDFS
Issue Type: Bug
Components: snapshots
Reporter: Shashikant Banerjee
This is the flow of steps where we see a discrepancy between REM_QUOTA and new
file operation failure. REM_QUOTA shows a value of 1 but file creation
operation does not succeed.
{code:java}
hdfs@c3265-node3 root$ hdfs dfs -mkdir /dir1
hdfs@c3265-node3 root$ hdfs dfsadmin -setQuota 2 /dir1
hdfs@c3265-node3 root$ hdfs dfsadmin -allowSnapshot /dir1
Allowing snaphot on /dir1 succeeded
hdfs@c3265-node3 root$ hdfs dfs -touchz /dir1/file1
hdfs@c3265-node3 root$ hdfs dfs -createSnapshot /dir1 snap1
Created snapshot /dir1/.snapshot/snap1
hdfs@c3265-node3 root$ hdfs dfs -count -v -q /dir1
QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE
PATHNAME
2 0 none inf 1 1 0 /dir1
hdfs@c3265-node3 root$ hdfs dfs -rm /dir1/file1
19/03/26 11:20:25 INFO fs.TrashPolicyDefault: Moved:
'hdfs://smajetinn/dir1/file1' to trash at:
hdfs://smajetinn/user/hdfs/.Trash/Current/dir1/file11553599225772
hdfs@c3265-node3 root$ hdfs dfs -count -v -q /dir1
QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE
PATHNAME
2 1 none inf 1 0 0 /dir1
hdfs@c3265-node3 root$ hdfs dfs -touchz /dir1/file1
touchz: The NameSpace quota (directories and files) of directory /dir1 is
exceeded: quota=2 file count=3{code}
The issue here, is that the count command takes only files and directories into
account not the inode references. When trash is enabled, the deletion of files
inside a directory actually does a rename operation as a result of which an
inode reference is maintained in the deleted list of the snapshot diff which is
taken into account while computing the namespace quota, but count command
(getContentSummary()) ,just takes into account just the files and directories,
not the referenced entity for calculating the REM_QUOTA. The referenced entity
is taken into account for space quota only.
InodeReference.java:
---
{code:java}
@Override
public final ContentSummaryComputationContext computeContentSummary(
int snapshotId, ContentSummaryComputationContext summary) {
final int s = snapshotId < lastSnapshotId ? snapshotId : lastSnapshotId;
// only count storagespace for WithName
final QuotaCounts q = computeQuotaUsage(
summary.getBlockStoragePolicySuite(), getStoragePolicyID(), false, s);
summary.getCounts().addContent(Content.DISKSPACE, q.getStorageSpace());
summary.getCounts().addTypeSpaces(q.getTypeSpaces());
return summary;
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14186) blockreport storm slow down namenode restart seriously in large cluster
[ https://issues.apache.org/jira/browse/HDFS-14186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842097#comment-16842097 ] He Xiaoqiao commented on HDFS-14186: [~xuzq_zander], Thanks for your suggestion, it is valid point, but I don't think it could solve this issue completely, because a. files have different replications, in my practice, it is between 2 and 10; b. if we config it but replicas of majority blocks is less than this value, it will never leave safemode, another way, even if we config a proper value, block storm could still occur as mentioned above. Thanks again. > blockreport storm slow down namenode restart seriously in large cluster > --- > > Key: HDFS-14186 > URL: https://issues.apache.org/jira/browse/HDFS-14186 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.1 >Reporter: He Xiaoqiao >Assignee: He Xiaoqiao >Priority: Major > Attachments: HDFS-14186.001.patch > > > In the current implementation, the datanode sends blockreport immediately > after register to namenode successfully when restart, and the blockreport > storm will make namenode high load to process them. One result is some > received RPC have to skip because queue time is timeout. If some datanodes' > heartbeat RPC are continually skipped for long times (default is > heartbeatExpireInterval=630s) it will be set DEAD, then datanode has to > re-register and send blockreport again, aggravate blockreport storm and trap > in a vicious circle, and slow down (more than one hour and even more) > namenode startup seriously in a large (several thousands of datanodes) and > busy cluster especially. Although there are many work to optimize namenode > startup, the issue still exists. > I propose to postpone dead datanode check when namenode have finished startup. > Any comments and suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1487) Bootstrap React framework for Recon UI
[ https://issues.apache.org/jira/browse/HDDS-1487?focusedWorklogId=243925=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243925 ] ASF GitHub Bot logged work on HDDS-1487: Author: ASF GitHub Bot Created on: 17/May/19 11:08 Start Date: 17/May/19 11:08 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #831: HDDS-1487. Bootstrap React framework for Recon UI URL: https://github.com/apache/hadoop/pull/831#issuecomment-493416133 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 31 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 398 | trunk passed | | +1 | compile | 202 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 1018 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 127 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 536 | the patch passed | | -1 | jshint | 2427 | The patch generated 15 new + 0 unchanged - 7061 fixed = 15 total (was 7061) | | +1 | compile | 264 | the patch passed | | +1 | javac | 264 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 622 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 141 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 138 | hadoop-hdds in the patch failed. | | -1 | unit | 1439 | hadoop-ozone in the patch failed. | | +1 | asflicense | 46 | The patch does not generate ASF License warnings. | | | | 7312 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion | | | hadoop.ozone.om.TestScmSafeMode | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-831/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/831 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml jshint | | uname | Linux eecc757b3f47 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / c183bd8 | | Default Java | 1.8.0_212 | | jshint | https://builds.apache.org/job/hadoop-multibranch/job/PR-831/1/artifact/out/diff-patch-jshint.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-831/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-831/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-831/1/testReport/ | | Max. process+thread count | 4563 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-recon U: hadoop-ozone/ozone-recon | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-831/1/console | | versions | git=2.7.4 maven=3.3.9 jshint=2.10.2 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243925) Time Spent: 20m (was: 10m) > Bootstrap React framework for Recon UI > -- > > Key: HDDS-1487 > URL: https://issues.apache.org/jira/browse/HDDS-1487 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Affects Versions: 0.4.0 >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate:
[jira] [Commented] (HDFS-14186) blockreport storm slow down namenode restart seriously in large cluster
[ https://issues.apache.org/jira/browse/HDFS-14186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842059#comment-16842059 ] xuzq commented on HDFS-14186: - Hi, [~hexiaoqiao], improve the dfs.namenode.replication.min can avoid this problem? > blockreport storm slow down namenode restart seriously in large cluster > --- > > Key: HDFS-14186 > URL: https://issues.apache.org/jira/browse/HDFS-14186 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.1 >Reporter: He Xiaoqiao >Assignee: He Xiaoqiao >Priority: Major > Attachments: HDFS-14186.001.patch > > > In the current implementation, the datanode sends blockreport immediately > after register to namenode successfully when restart, and the blockreport > storm will make namenode high load to process them. One result is some > received RPC have to skip because queue time is timeout. If some datanodes' > heartbeat RPC are continually skipped for long times (default is > heartbeatExpireInterval=630s) it will be set DEAD, then datanode has to > re-register and send blockreport again, aggravate blockreport storm and trap > in a vicious circle, and slow down (more than one hour and even more) > namenode startup seriously in a large (several thousands of datanodes) and > busy cluster especially. Although there are many work to optimize namenode > startup, the issue still exists. > I propose to postpone dead datanode check when namenode have finished startup. > Any comments and suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14402) Use FileChannel.transferTo() method for transferring block to SCM cache
[ https://issues.apache.org/jira/browse/HDFS-14402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842054#comment-16842054 ] Feilong He commented on HDFS-14402: --- Thanks [~rakeshr] for your valuable comments. The checksum logic for pmem cache is consistent with that for DRAM cache. As you pointed, it looks more reasonable that the checksum should be executed for cache data already on DRAM/Pmem, instead of for original data on disk. I will consider your comment. Upon making checksum configurable, I am thinking maybe to user, the cache read performance is concerned mostly. The checksum is executed once when caching data to DRAM/Pmem. It may be tolerable to user with checksum operation for data verification. I think more discussions are required. I will open a separate Jira common to DRAM and Pmem cache. If DRAM and Pmem cache have respective checksum logic in our future impl, #fileBuffer should be kept protected. Thanks again! > Use FileChannel.transferTo() method for transferring block to SCM cache > --- > > Key: HDFS-14402 > URL: https://issues.apache.org/jira/browse/HDFS-14402 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, datanode >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Attachments: HDFS-14402.000.patch, HFDS-14402.001.patch > > > We will consider to use transferTo API to improve SCM's cach performace. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14498) LeaseManager can loop forever on the file for which create has failed
[
https://issues.apache.org/jira/browse/HDFS-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842050#comment-16842050
]
xuzq commented on HDFS-14498:
-
Hi, [~sershe]. It seem to be one bug.
Can you provide the method to reproduce this problem?
Maybe we can skip this expired lease to avoid blocking other expired leases,
and recheck it at the next checkLeases().
> LeaseManager can loop forever on the file for which create has failed
> --
>
> Key: HDFS-14498
> URL: https://issues.apache.org/jira/browse/HDFS-14498
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Sergey Shelukhin
>Priority: Major
>
> The logs from file creation are long gone due to infinite lease logging,
> however it presumably failed... the client who was trying to write this file
> is definitely long dead.
> The version includes HDFS-4882.
> We get this log pattern repeating infinitely:
> {noformat}
> 2019-05-16 14:00:16,893 INFO
> [org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@b27557f]
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: [Lease. Holder:
> DFSClient_NONMAPREDUCE_-20898906_61, pending creates: 1] has expired hard
> limit
> 2019-05-16 14:00:16,893 INFO
> [org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@b27557f]
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering [Lease.
> Holder: DFSClient_NONMAPREDUCE_-20898906_61, pending creates: 1], src=
> 2019-05-16 14:00:16,893 WARN
> [org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@b27557f]
> org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.internalReleaseLease:
> Failed to release lease for file . Committed blocks are waiting to be
> minimally replicated. Try again later.
> 2019-05-16 14:00:16,893 WARN
> [org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@b27557f]
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: Cannot release the path
> in the lease [Lease. Holder: DFSClient_NONMAPREDUCE_-20898906_61,
> pending creates: 1]. It will be retried.
> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: DIR*
> NameSystem.internalReleaseLease: Failed to release lease for file .
> Committed blocks are waiting to be minimally replicated. Try again later.
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.internalReleaseLease(FSNamesystem.java:3357)
> at
> org.apache.hadoop.hdfs.server.namenode.LeaseManager.checkLeases(LeaseManager.java:573)
> at
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:509)
> at java.lang.Thread.run(Thread.java:745)
> $ grep -c "Recovering.*DFSClient_NONMAPREDUCE_-20898906_61, pending creates:
> 1" hdfs_nn*
> hdfs_nn.log:1068035
> hdfs_nn.log.2019-05-16-14:1516179
> hdfs_nn.log.2019-05-16-15:1538350
> {noformat}
> Aside from an actual bug fix, it might make sense to make LeaseManager not
> log so much, in case if there are more bugs like this...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1553) Add metrics in rack aware container placement policy
Sammi Chen created HDDS-1553: Summary: Add metrics in rack aware container placement policy Key: HDDS-1553 URL: https://issues.apache.org/jira/browse/HDDS-1553 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Sammi Chen To collect following statistics, 1. total requested datanode count (A) 2. success allocated datanode count without constrain compromise (B) 3. success allocated datanode count with some comstrain compromise (C) 4. failed to allocated datanode count (D) A = B + C + D -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1525) Mapreduce failure when using Hadoop 2.7.5
[ https://issues.apache.org/jira/browse/HDDS-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842026#comment-16842026 ] Sammi Chen edited comment on HDDS-1525 at 5/17/19 9:26 AM: --- [~xyao], right, the root cause is class KeyProviderTokenIssuer is not available in Hadoop 2. To address this issue, we can provide a new RpcClient for Hadoop 2. [~xyao] what do you think of this solution? Fire a new JIRA HDDS-1552 for this task. was (Author: sammi): [~xyao], right, the root cause is class KeyProviderTokenIssuer is not available in Hadoop 2. To address this issue, we can provide a new RpcClient for Hadoop 2. [~xyao] what do you think of this solution? > Mapreduce failure when using Hadoop 2.7.5 > - > > Key: HDDS-1525 > URL: https://issues.apache.org/jira/browse/HDDS-1525 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Affects Versions: 0.4.0 >Reporter: Sammi Chen >Priority: Major > Attachments: teragen.log > > > Integrate Ozone(0.4 branch) with Hadoop 2.7.5, "hdfs dfs -ls /" can pass, > while teragen failed. > When add -verbose:class to java options, it shows that class KeyProvider is > loaded twice by different classloader while it is only loaded once when > execute "hdfs dfs -ls /" > All jars under share/ozone/lib are added into hadoop classpath except ozone > file system current lib jar. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1552) Provide RpcClient for Hadoop 2
Sammi Chen created HDDS-1552: Summary: Provide RpcClient for Hadoop 2 Key: HDDS-1552 URL: https://issues.apache.org/jira/browse/HDDS-1552 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Sammi Chen Provide a RpcClient for Hadoop 2. Current RpcClient depends on class KeyProviderTokenIssuer which is not available in Hadoop 2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1525) Mapreduce failure when using Hadoop 2.7.5
[ https://issues.apache.org/jira/browse/HDDS-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842026#comment-16842026 ] Sammi Chen edited comment on HDDS-1525 at 5/17/19 9:24 AM: --- [~xyao], right, the root cause is class KeyProviderTokenIssuer is not available in Hadoop 2. To address this issue, we can provide a new RpcClient for Hadoop 2. [~xyao] what do you think of this solution? was (Author: sammi): [~xyao], right, the root cause is class KeyProviderTokenIssuer is not available in Hadoop 2. To address this issue, we can provide a new RpcClient for Hadoop 2. [~xyao] waht do you think of this solution? > Mapreduce failure when using Hadoop 2.7.5 > - > > Key: HDDS-1525 > URL: https://issues.apache.org/jira/browse/HDDS-1525 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Affects Versions: 0.4.0 >Reporter: Sammi Chen >Priority: Major > Attachments: teragen.log > > > Integrate Ozone(0.4 branch) with Hadoop 2.7.5, "hdfs dfs -ls /" can pass, > while teragen failed. > When add -verbose:class to java options, it shows that class KeyProvider is > loaded twice by different classloader while it is only loaded once when > execute "hdfs dfs -ls /" > All jars under share/ozone/lib are added into hadoop classpath except ozone > file system current lib jar. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1525) Mapreduce failure when using Hadoop 2.7.5
[ https://issues.apache.org/jira/browse/HDDS-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842026#comment-16842026 ] Sammi Chen commented on HDDS-1525: -- [~xyao], right, the root cause is class KeyProviderTokenIssuer is not available in Hadoop 2. To address this issue, we can provide a new RpcClient for Hadoop 2. [~xyao] do you think of this solution? > Mapreduce failure when using Hadoop 2.7.5 > - > > Key: HDDS-1525 > URL: https://issues.apache.org/jira/browse/HDDS-1525 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Affects Versions: 0.4.0 >Reporter: Sammi Chen >Priority: Major > Attachments: teragen.log > > > Integrate Ozone(0.4 branch) with Hadoop 2.7.5, "hdfs dfs -ls /" can pass, > while teragen failed. > When add -verbose:class to java options, it shows that class KeyProvider is > loaded twice by different classloader while it is only loaded once when > execute "hdfs dfs -ls /" > All jars under share/ozone/lib are added into hadoop classpath except ozone > file system current lib jar. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1525) Mapreduce failure when using Hadoop 2.7.5
[ https://issues.apache.org/jira/browse/HDDS-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842026#comment-16842026 ] Sammi Chen edited comment on HDDS-1525 at 5/17/19 9:11 AM: --- [~xyao], right, the root cause is class KeyProviderTokenIssuer is not available in Hadoop 2. To address this issue, we can provide a new RpcClient for Hadoop 2. [~xyao] waht do you think of this solution? was (Author: sammi): [~xyao], right, the root cause is class KeyProviderTokenIssuer is not available in Hadoop 2. To address this issue, we can provide a new RpcClient for Hadoop 2. [~xyao] do you think of this solution? > Mapreduce failure when using Hadoop 2.7.5 > - > > Key: HDDS-1525 > URL: https://issues.apache.org/jira/browse/HDDS-1525 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Affects Versions: 0.4.0 >Reporter: Sammi Chen >Priority: Major > Attachments: teragen.log > > > Integrate Ozone(0.4 branch) with Hadoop 2.7.5, "hdfs dfs -ls /" can pass, > while teragen failed. > When add -verbose:class to java options, it shows that class KeyProvider is > loaded twice by different classloader while it is only loaded once when > execute "hdfs dfs -ls /" > All jars under share/ozone/lib are added into hadoop classpath except ozone > file system current lib jar. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1487) Bootstrap React framework for Recon UI
[ https://issues.apache.org/jira/browse/HDDS-1487?focusedWorklogId=243891=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243891 ] ASF GitHub Bot logged work on HDDS-1487: Author: ASF GitHub Bot Created on: 17/May/19 09:05 Start Date: 17/May/19 09:05 Worklog Time Spent: 10m Work Description: vivekratnavel commented on pull request #831: HDDS-1487. Bootstrap React framework for Recon UI URL: https://github.com/apache/hadoop/pull/831 Bootstrap React with Typescript, Ant, LESS and other necessary libraries for Recon UI. This PR also adds necessary changes to recon pom.xml to bring React UI up when starting recon server This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243891) Time Spent: 10m Remaining Estimate: 0h > Bootstrap React framework for Recon UI > -- > > Key: HDDS-1487 > URL: https://issues.apache.org/jira/browse/HDDS-1487 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Affects Versions: 0.4.0 >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Bootstrap React with Typescript, Ant, LESS and other necessary libraries for > Recon UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1487) Bootstrap React framework for Recon UI
[ https://issues.apache.org/jira/browse/HDDS-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1487: - Labels: pull-request-available (was: ) > Bootstrap React framework for Recon UI > -- > > Key: HDDS-1487 > URL: https://issues.apache.org/jira/browse/HDDS-1487 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Affects Versions: 0.4.0 >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: pull-request-available > > Bootstrap React with Typescript, Ant, LESS and other necessary libraries for > Recon UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?focusedWorklogId=243888=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243888 ] ASF GitHub Bot logged work on HDDS-1530: Author: ASF GitHub Bot Created on: 17/May/19 08:31 Start Date: 17/May/19 08:31 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #830: HDDS-1530. Freon support big files larger than 2GB and add --bufferSize and --validateWrites options. URL: https://github.com/apache/hadoop/pull/830#issuecomment-493370392 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 41 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 427 | trunk passed | | +1 | compile | 196 | trunk passed | | +1 | checkstyle | 52 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 890 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 145 | trunk passed | | 0 | spotbugs | 271 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 478 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 407 | the patch passed | | +1 | compile | 204 | the patch passed | | +1 | javac | 204 | the patch passed | | +1 | checkstyle | 55 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 696 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 122 | the patch passed | | +1 | findbugs | 441 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 165 | hadoop-hdds in the patch failed. | | -1 | unit | 1189 | hadoop-ozone in the patch failed. | | +1 | asflicense | 37 | The patch does not generate ASF License warnings. | | | | 5673 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.TestContainerReplication | | | hadoop.ozone.scm.node.TestQueryNode | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-830/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/830 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 1c3367a285ee 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / c183bd8 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-830/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-830/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-830/1/testReport/ | | Max. process+thread count | 5268 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/tools U: hadoop-ozone/tools | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-830/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243888) Time Spent: 20m (was: 10m) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > *Current problems:* > 1. Freon does not
[jira] [Commented] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[
https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841983#comment-16841983
]
Hadoop QA commented on HDDS-1530:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
7s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 4m
15s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
42s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 56s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 54s{color}
| {color:red} hadoop-hdds in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 11s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.client.rpc.TestWatchForCommit |
| | hadoop.hdds.scm.pipeline.TestSCMPipelineManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/PreCommit-HDDS-Build/2697/artifact/out/Dockerfile
|
| JIRA Issue | HDDS-1530 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12968987/HDDS-1530.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 3f2ab20125b2 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb
13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / c183bd8 |
| Default Java | 1.8.0_212 |
| unit |
