[jira] [Assigned] (HDDS-1200) Ozone Data Scrubbing : Checksum verification for chunks
[ https://issues.apache.org/jira/browse/HDDS-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Supratim Deka reassigned HDDS-1200: --- Assignee: (was: Supratim Deka) > Ozone Data Scrubbing : Checksum verification for chunks > --- > > Key: HDDS-1200 > URL: https://issues.apache.org/jira/browse/HDDS-1200 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Supratim Deka >Priority: Major > > Background scrubber should read each chunk and verify the checksum. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1337) Handle GroupMismatchException in OzoneClient
[
https://issues.apache.org/jira/browse/HDDS-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807390#comment-16807390
]
Yiqun Lin edited comment on HDDS-1337 at 4/2/19 5:25 AM:
-
Hi [~shashikant], checkstyle issue and failure tests (
TestContainerStateMachineFailures/ TestContainerStateMachineFailures)is
related. Could you please have a look?
The stack:
{noformat}
java.io.IOException: Unexpected Storage Container Exception:
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
Requested operation not allowed as ContainerState is UNHEALTHY
at
org.apache.hadoop.hdds.scm.storage.BlockOutputStream.setIoException(BlockOutputStream.java:628)
at
org.apache.hadoop.hdds.scm.storage.BlockOutputStream.validateResponse(BlockOutputStream.java:619)
at
org.apache.hadoop.hdds.scm.storage.BlockOutputStream.lambda$writeChunkToContainer$6(BlockOutputStream.java:700)
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
was (Author: linyiqun):
Hi [~shashikant], checkstyle issue and failure tests (
TestContainerStateMachineFailures/ TestContainerStateMachineFailures)is
related. Can you have a look?
> Handle GroupMismatchException in OzoneClient
>
>
> Key: HDDS-1337
> URL: https://issues.apache.org/jira/browse/HDDS-1337
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Client
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Blocker
> Labels: Blocker
> Fix For: 0.4.0
>
> Attachments: HDDS-1337.000.patch, HDDS-1337.001.patch,
> HDDS-1337.002.patch, HDDS-1337.004.patch
>
>
> If a pipeline gets destroyed in ozone client, ozone client may hit
> GroupMismatchException from Ratis. In cases as such, client should exclude
> the pipeline and retry write to a different block.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1365) Fix error handling in KeyValueContainerCheck
[ https://issues.apache.org/jira/browse/HDDS-1365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Supratim Deka updated HDDS-1365: Attachment: HDDS-1365.000.patch Status: Patch Available (was: Open) > Fix error handling in KeyValueContainerCheck > > > Key: HDDS-1365 > URL: https://issues.apache.org/jira/browse/HDDS-1365 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Datanode >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > Attachments: HDDS-1365.000.patch > > > Error handling and propagation in KeyValueContainerCheck needs to be based on > throwing IOException instead of passing an error code to the calling function. > HDDS-1163 implemented the basic framework using a mix of error code return > and exception handling. There is added complexity because exceptions deep > inside the call chain are being caught and translated to error code return > values. The goal is to change all error handling in this class to use > Exceptions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1337) Handle GroupMismatchException in OzoneClient
[ https://issues.apache.org/jira/browse/HDDS-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807390#comment-16807390 ] Yiqun Lin commented on HDDS-1337: - Hi [~shashikant], checkstyle issue and failure tests ( TestContainerStateMachineFailures/ TestContainerStateMachineFailures)is related. Can you have a look? > Handle GroupMismatchException in OzoneClient > > > Key: HDDS-1337 > URL: https://issues.apache.org/jira/browse/HDDS-1337 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Blocker > Labels: Blocker > Fix For: 0.4.0 > > Attachments: HDDS-1337.000.patch, HDDS-1337.001.patch, > HDDS-1337.002.patch, HDDS-1337.004.patch > > > If a pipeline gets destroyed in ozone client, ozone client may hit > GroupMismatchException from Ratis. In cases as such, client should exclude > the pipeline and retry write to a different block. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-3246) pRead equivalent for direct read path
[
https://issues.apache.org/jira/browse/HDFS-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807375#comment-16807375
]
Hadoop QA commented on HDFS-3246:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdfs-project/hadoop-hdfs-native-client {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
59s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 4s{color} | {color:orange} root: The patch generated 3 new + 112 unchanged
- 7 fixed = 115 total (was 119) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdfs-project/hadoop-hdfs-native-client {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
59s{color} | {color:red} hadoop-common-project_hadoop-common generated 2 new +
0 unchanged - 0 fixed = 2 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
23s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
58s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}113m 9s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
32s{color} | {color:green} hadoop-hdfs-native-client in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} |
[jira] [Commented] (HDFS-14394) Add -std=c99 / -std=gnu99 to libhdfs compile flags
[
https://issues.apache.org/jira/browse/HDFS-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807365#comment-16807365
]
Sahil Takiar commented on HDFS-14394:
-
[~eyang] do those tests pass locally on trunk without the patch posted here?
I've seen some errors like that before and it usually is environmental. Hadoop
QA was able to build the patch, but I don't think it ran the libhdfs++ tests.
I'm not sure I understand that link, but I don't think adding {{-std=c99}} to
{{CMAKE_C_FLAGS}} forces all libhdfs++ code to be compiled using C99. My
understanding is that {{CMAKE_C_FLAGS}} is for compiling C files and
{{CMAKE_CXX_FLAGS}} is for compiling C++ files. The libhdfs++ CMake file
already includes {{-std=c++11}}. There are a few of C files in libhdfs++, but
most seem to be testing related.
> Add -std=c99 / -std=gnu99 to libhdfs compile flags
> --
>
> Key: HDFS-14394
> URL: https://issues.apache.org/jira/browse/HDFS-14394
> Project: Hadoop HDFS
> Issue Type: Task
> Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HDFS-14394.001.patch
>
>
> libhdfs compilation currently does not enforce a minimum required C version.
> As of today, the libhdfs build on Hadoop QA works, but when built on a
> machine with an outdated gcc / cc version where C89 is the default,
> compilation fails due to errors such as:
> {code}
> /build/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jclasses.c:106:5:
> error: ‘for’ loop initial declarations are only allowed in C99 mode
> for (int i = 0; i < numCachedClasses; i++) {
> ^
> /build/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jclasses.c:106:5:
> note: use option -std=c99 or -std=gnu99 to compile your code
> {code}
> We should add the -std=c99 / -std=gnu99 flags to libhdfs compilation so that
> we can enforce C99 as the minimum required version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13972) RBF: Support for Delegation Token (WebHDFS)
[
https://issues.apache.org/jira/browse/HDFS-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807361#comment-16807361
]
Hadoop QA commented on HDFS-13972:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
5s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 22m
25s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-13972 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964506/HDFS-13972-HDFS-13891.011.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux aa3a1a9c86ba 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / dea3798 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26565/testReport/ |
| Max. process+thread count | 1414 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26565/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Support for Delegation Token (WebHDFS)
> ---
>
> Key:
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[
https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807341#comment-16807341
]
star commented on HDFS-14370:
-
[~xkrogen]， agree. Server side handler as limited resource should not be
blocked.
{quote}This is similar to how the NN, when it wants a client to backoff due to
load, throws a backoff exception and expects the client to act accordingly.
{quote}
> Edit log tailing fast-path should allow for backoff
> ---
>
> Key: HDFS-14370
> URL: https://issues.apache.org/jira/browse/HDFS-14370
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, qjm
>Affects Versions: 3.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
>
> As part of HDFS-13150, in-progress edit log tailing was changed to use an
> RPC-based mechanism, thus allowing the edit log tailing frequency to be
> turned way down, and allowing standby/observer NameNodes to be only a few
> milliseconds stale as compared to the Active NameNode.
> When there is a high volume of transactions on the system, each RPC fetches
> transactions and takes some time to process them, self-rate-limiting how
> frequently an RPC is submitted. In a lightly loaded cluster, however, most of
> these RPCs return an empty set of transactions, consuming a high
> (de)serialization overhead for very little benefit. This was reported by
> [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where
> the SbNN was submitting 8000 RPCs per second that returned empty.
> I propose we add some sort of backoff to the tailing, so that if an empty
> response is received, it will wait a longer period of time before submitting
> a new RPC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1355) Only FQDN is accepted for OM rpc address in secure environment
[ https://issues.apache.org/jira/browse/HDDS-1355?focusedWorklogId=221580=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221580 ] ASF GitHub Bot logged work on HDDS-1355: Author: ASF GitHub Bot Created on: 02/Apr/19 01:35 Start Date: 02/Apr/19 01:35 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #677: HDDS-1355. Only FQDN is accepted for OM rpc address in secure environment. Contributed by Ajay Kumar. URL: https://github.com/apache/hadoop/pull/677#issuecomment-478809174 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 25 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 57 | Maven dependency ordering for branch | | +1 | mvninstall | 1091 | trunk passed | | +1 | compile | 104 | trunk passed | | +1 | checkstyle | 28 | trunk passed | | +1 | mvnsite | 75 | trunk passed | | +1 | shadedclient | 758 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 46 | trunk passed | | +1 | javadoc | 46 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 13 | Maven dependency ordering for patch | | +1 | mvninstall | 70 | the patch passed | | +1 | compile | 93 | the patch passed | | +1 | javac | 93 | the patch passed | | +1 | checkstyle | 21 | the patch passed | | +1 | mvnsite | 52 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 690 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 43 | the patch passed | | +1 | javadoc | 30 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 39 | ozone-manager in the patch passed. | | -1 | unit | 1078 | integration-test in the patch failed. | | +1 | asflicense | 29 | The patch does not generate ASF License warnings. | | | | 4439 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.TestMiniChaosOzoneCluster | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-677/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/677 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux a86d61a09e9b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / ab2bda5 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-677/1/artifact/out/patch-unit-hadoop-ozone_integration-test.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-677/1/testReport/ | | Max. process+thread count | 4992 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-677/1/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 221580) Time Spent: 20m (was: 10m) > Only FQDN is accepted for OM rpc address in secure environment > -- > > Key: HDDS-1355 > URL: https://issues.apache.org/jira/browse/HDDS-1355 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Ajay Kumar >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > While the scm address can be a host name (relative to the current search > domain)
[jira] [Work logged] (HDDS-1324) TestOzoneManagerHA seems to be flaky
[
https://issues.apache.org/jira/browse/HDDS-1324?focusedWorklogId=221555=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221555
]
ASF GitHub Bot logged work on HDDS-1324:
Author: ASF GitHub Bot
Created on: 02/Apr/19 01:03
Start Date: 02/Apr/19 01:03
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on issue #676: HDDS-1324.
TestOzoneManagerHA tests are flaky
URL: https://github.com/apache/hadoop/pull/676#issuecomment-478801969
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|::|--:|:|:|
| 0 | reexec | 25 | Docker mode activated. |
||| _ Prechecks _ |
| +1 | @author | 0 | The patch does not contain any @author tags. |
| +1 | test4tests | 0 | The patch appears to include 1 new or modified test
files. |
||| _ trunk Compile Tests _ |
| +1 | mvninstall | 1015 | trunk passed |
| +1 | compile | 45 | trunk passed |
| +1 | checkstyle | 20 | trunk passed |
| +1 | mvnsite | 32 | trunk passed |
| +1 | shadedclient | 721 | branch has no errors when building and testing
our client artifacts. |
| 0 | findbugs | 0 | Skipped patched modules with no Java source:
hadoop-ozone/integration-test |
| +1 | findbugs | 0 | trunk passed |
| +1 | javadoc | 20 | trunk passed |
||| _ Patch Compile Tests _ |
| +1 | mvninstall | 32 | the patch passed |
| +1 | compile | 24 | the patch passed |
| +1 | javac | 24 | the patch passed |
| +1 | checkstyle | 16 | the patch passed |
| +1 | mvnsite | 26 | the patch passed |
| +1 | whitespace | 0 | The patch has no whitespace issues. |
| +1 | shadedclient | 735 | patch has no errors when building and testing
our client artifacts. |
| 0 | findbugs | 0 | Skipped patched modules with no Java source:
hadoop-ozone/integration-test |
| +1 | findbugs | 0 | the patch passed |
| +1 | javadoc | 17 | the patch passed |
||| _ Other Tests _ |
| -1 | unit | 742 | integration-test in the patch failed. |
| +1 | asflicense | 37 | The patch does not generate ASF License warnings. |
| | | 3589 | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.ozone.web.client.TestKeysRatis |
| | hadoop.ozone.TestStorageContainerManager |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-676/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/676 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 889ec222af34 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ab2bda5 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| unit |
https://builds.apache.org/job/hadoop-multibranch/job/PR-676/2/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
|
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-676/2/testReport/ |
| Max. process+thread count | 4655 (vs. ulimit of 5500) |
| modules | C: hadoop-ozone/integration-test U:
hadoop-ozone/integration-test |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-676/2/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 221555)
Time Spent: 0.5h (was: 20m)
> TestOzoneManagerHA seems to be flaky
>
>
> Key: HDDS-1324
> URL: https://issues.apache.org/jira/browse/HDDS-1324
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Affects Versions: 0.5.0
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> TestOzoneManagerHA failed once with the following error:
> {code}
> [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 105.931 s <<< FAILURE! - in org.apache.hadoop.ozone.om.TestOzoneManagerHA
> [ERROR] testOMRetryProxy(org.apache.hadoop.ozone.om.TestOzoneManagerHA) Time
>
[jira] [Updated] (HDFS-13972) RBF: Support for Delegation Token (WebHDFS)
[ https://issues.apache.org/jira/browse/HDFS-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CR Hota updated HDFS-13972: --- Attachment: HDFS-13972-HDFS-13891.011.patch > RBF: Support for Delegation Token (WebHDFS) > --- > > Key: HDFS-13972 > URL: https://issues.apache.org/jira/browse/HDFS-13972 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: HDFS-13972-HDFS-13891.001.patch, > HDFS-13972-HDFS-13891.002.patch, HDFS-13972-HDFS-13891.003.patch, > HDFS-13972-HDFS-13891.004.patch, HDFS-13972-HDFS-13891.005.patch, > HDFS-13972-HDFS-13891.006.patch, HDFS-13972-HDFS-13891.007.patch, > HDFS-13972-HDFS-13891.008.patch, HDFS-13972-HDFS-13891.009.patch, > HDFS-13972-HDFS-13891.010.patch, HDFS-13972-HDFS-13891.011.patch, > TestRouterWebHDFSContractTokens.java > > > HDFS Router should support issuing HDFS delegation tokens through WebHDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14400) Namenode ExpiredHeartbeats metric
[
https://issues.apache.org/jira/browse/HDFS-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807331#comment-16807331
]
Hadoop QA commented on HDFS-14400:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 46s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 19s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}170m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14400 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964490/HDFS-14400-003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 4ec8b5a76bb2 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ab2bda5 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26564/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26564/testReport/ |
| Max. process+thread count | 2939 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
|
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807311#comment-16807311
]
Konstantin Shvachko commented on HDFS-10477:
Let's target this for 2.10 at the minimum.
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.004.patch, HDFS-10477.005.patch, HDFS-10477.006.patch,
> HDFS-10477.007.patch, HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 280219 over-replicated blocks on 10.142.27.14:1004 during recommissioning
>
[jira] [Updated] (HDDS-1355) Only FQDN is accepted for OM rpc address in secure environment
[
https://issues.apache.org/jira/browse/HDDS-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-1355:
-
Labels: pull-request-available (was: )
> Only FQDN is accepted for OM rpc address in secure environment
> --
>
> Key: HDDS-1355
> URL: https://issues.apache.org/jira/browse/HDDS-1355
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Ajay Kumar
>Priority: Major
> Labels: pull-request-available
>
> While the scm address can be a host name (relative to the current search
> domain) if the om address is just a hostname and not a FQDN a NPE is thrown:
> {code}
> 10 │ OZONE-SITE.XML_ozone.om.address: "om-0.om"
> 11 │ OZONE-SITE.XML_ozone.scm.client.address: "scm-0.scm"
> 12 │ OZONE-SITE.XML_ozone.scm.names: "scm-0.scm"
> {code}
> {code}
> 2019-03-29 14:37:52 ERROR OzoneManager:865 - Failed to start the OzoneManager.
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.om.OzoneManager.getSCMSignedCert(OzoneManager.java:1372)
> at
> org.apache.hadoop.ozone.om.OzoneManager.initializeSecurity(OzoneManager.java:1018)
> at org.apache.hadoop.ozone.om.OzoneManager.omInit(OzoneManager.java:971)
> at org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:928)
> at org.apache.hadoop.ozone.om.OzoneManager.main(OzoneManager.java:859)
> {code}
> I don't know what is the right validation rule here, but I am pretty sure
> that NPE should be avoided and a meaningful error should be thrown. (and the
> behaviour should be the same for scm and om)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1355) Only FQDN is accepted for OM rpc address in secure environment
[
https://issues.apache.org/jira/browse/HDDS-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HDDS-1355:
-
Status: Patch Available (was: Open)
> Only FQDN is accepted for OM rpc address in secure environment
> --
>
> Key: HDDS-1355
> URL: https://issues.apache.org/jira/browse/HDDS-1355
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Ajay Kumar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While the scm address can be a host name (relative to the current search
> domain) if the om address is just a hostname and not a FQDN a NPE is thrown:
> {code}
> 10 │ OZONE-SITE.XML_ozone.om.address: "om-0.om"
> 11 │ OZONE-SITE.XML_ozone.scm.client.address: "scm-0.scm"
> 12 │ OZONE-SITE.XML_ozone.scm.names: "scm-0.scm"
> {code}
> {code}
> 2019-03-29 14:37:52 ERROR OzoneManager:865 - Failed to start the OzoneManager.
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.om.OzoneManager.getSCMSignedCert(OzoneManager.java:1372)
> at
> org.apache.hadoop.ozone.om.OzoneManager.initializeSecurity(OzoneManager.java:1018)
> at org.apache.hadoop.ozone.om.OzoneManager.omInit(OzoneManager.java:971)
> at org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:928)
> at org.apache.hadoop.ozone.om.OzoneManager.main(OzoneManager.java:859)
> {code}
> I don't know what is the right validation rule here, but I am pretty sure
> that NPE should be avoided and a meaningful error should be thrown. (and the
> behaviour should be the same for scm and om)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1355) Only FQDN is accepted for OM rpc address in secure environment
[
https://issues.apache.org/jira/browse/HDDS-1355?focusedWorklogId=221552=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221552
]
ASF GitHub Bot logged work on HDDS-1355:
Author: ASF GitHub Bot
Created on: 02/Apr/19 00:20
Start Date: 02/Apr/19 00:20
Worklog Time Spent: 10m
Work Description: ajayydv commented on pull request #677: HDDS-1355. Only
FQDN is accepted for OM rpc address in secure environment. Contributed by Ajay
Kumar.
URL: https://github.com/apache/hadoop/pull/677
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 221552)
Time Spent: 10m
Remaining Estimate: 0h
> Only FQDN is accepted for OM rpc address in secure environment
> --
>
> Key: HDDS-1355
> URL: https://issues.apache.org/jira/browse/HDDS-1355
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Ajay Kumar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While the scm address can be a host name (relative to the current search
> domain) if the om address is just a hostname and not a FQDN a NPE is thrown:
> {code}
> 10 │ OZONE-SITE.XML_ozone.om.address: "om-0.om"
> 11 │ OZONE-SITE.XML_ozone.scm.client.address: "scm-0.scm"
> 12 │ OZONE-SITE.XML_ozone.scm.names: "scm-0.scm"
> {code}
> {code}
> 2019-03-29 14:37:52 ERROR OzoneManager:865 - Failed to start the OzoneManager.
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.om.OzoneManager.getSCMSignedCert(OzoneManager.java:1372)
> at
> org.apache.hadoop.ozone.om.OzoneManager.initializeSecurity(OzoneManager.java:1018)
> at org.apache.hadoop.ozone.om.OzoneManager.omInit(OzoneManager.java:971)
> at org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:928)
> at org.apache.hadoop.ozone.om.OzoneManager.main(OzoneManager.java:859)
> {code}
> I don't know what is the right validation rule here, but I am pretty sure
> that NPE should be avoided and a meaningful error should be thrown. (and the
> behaviour should be the same for scm and om)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-10477:
---
Target Version/s: 2.10.0
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.004.patch, HDFS-10477.005.patch, HDFS-10477.006.patch,
> HDFS-10477.007.patch, HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 280219 over-replicated blocks on 10.142.27.14:1004 during recommissioning
> 2016-05-26 20:13:25,370 INFO
>
[jira] [Comment Edited] (HDFS-14394) Add -std=c99 / -std=gnu99 to libhdfs compile flags
[
https://issues.apache.org/jira/browse/HDFS-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807296#comment-16807296
]
Eric Yang edited comment on HDFS-14394 at 4/2/19 12:00 AM:
---
[~stakiar] It seems a little strange that it failed the first time, then it
worked the second time. The last section of the maven output looks like this:
{code:java}
[exec]
/home/eyang/test/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfspp_mini_dfs.h:124:
Failure
[exec] Expected: (nullptr) != (clusterInfo), actual: 8-byte object <00-00
00-00 00-00 00-00> vs NULL
[exec] #
[exec] # A fatal error has been detected by the Java Runtime Environment:
[exec] #
[exec] # SIGSEGV (0xb) at pc=0x00783b7e, pid=28414,
tid=0x7efc91ac38c0
[exec] #
[exec] # JRE version: OpenJDK Runtime Environment (8.0_151-b12) (build
1.8.0_151-b12)
[exec] # Java VM: OpenJDK 64-Bit Server VM (25.151-b12 mixed mode
linux-amd64 compressed oops)
[exec] # Problematic frame:
[exec] nmdCreate: Builder#build error:
[exec] RuntimeException: Although a UNIX domain socket path is configured
as /tmp/native_mini_dfs.sock.28414.846930886, we cannot start a
localDataXceiverServer because libhadoop cannot be
loaded.java.lang.RuntimeException: Although a UNIX domain socket path is
configured as /tmp/native_mini_dfs.sock.28414.846930886, we cannot start a
localDataXceiverServer because libhadoop cannot be loaded.
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:1209)
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:1178)
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1433)
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:509)
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2827)
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2733)
[exec] at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1697)
[exec] at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:913)
[exec] at
org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:520)
[exec] at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:479)
[exec] # C [hdfs_ext_hdfspp_test_shim_static+0x383b7e]
[exec] #
[exec] # Failed to write core dump. Core dumps have been disabled. To
enable core dumping, try "ulimit -c unlimited" before starting Java again
[exec] #
[exec] # An error report file with more information is saved as:
[exec] #
/home/eyang/test/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfspp/tests/hs_err_pid28414.log
[exec] #
[exec] # If you would like to submit a bug report, please visit:
[exec] # http://bugreport.java.com/bugreport/crash.jsp
[exec] # The crash happened outside the Java Virtual Machine in native
code.
[exec] # See problematic frame for where to report the bug.
[exec] #
[exec]
[exec]
[exec] 85% tests passed, 6 tests failed out of 40
[exec]
[exec] Total Test time (real) = 96.60 sec
[exec]
[exec] The following tests FAILED:
[exec] 3 - test_test_libhdfs_zerocopy_hdfs_static (Failed)
[exec] 36 - test_hdfspp_mini_dfs_smoke_hdfspp_test_shim_static
(OTHER_FAULT)
[exec] 37 - libhdfs_mini_stress_valgrind_hdfspp_test_static (Failed)
[exec] 38 - memcheck_libhdfs_mini_stress_valgrind_hdfspp_test_static
(Failed)
[exec] 39 - test_libhdfs_mini_stress_hdfspp_test_shim_static (Failed)
[exec] 40 - test_hdfs_ext_hdfspp_test_shim_static (OTHER_FAULT)
[exec] Errors while running CTest{code}
It was not able to find libhadoop native library, even though I did ran with
mvn clean install -Pnative in hadoop-common-project follow by the same compile
command in hadoop-hdfs-native-client project.
There are some difference between [c++11 and
c99|https://stackoverflow.com/questions/10461331/what-are-the-incompatible-differences-between-c99-and-c11].
I don't know if this will create instability in libhdfspp to use c99. Gcc
4.5+ can enforce c99 standard by passing -std=c99 -pedantic-errors
-fextended-identifiers instead of gnu99 to prevent gnu features to be included.
was (Author: eyang):
[~stakiar] It seems a little strange that it failed the first time, then it
worked the second time. The last section of the maven output looks like this:
{code:java}
[exec]
/home/eyang/test/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfspp_mini_dfs.h:124:
Failure
[exec] Expected: (nullptr) != (clusterInfo), actual: 8-byte object <00-00
[jira] [Commented] (HDFS-14369) RBF: Fix trailing "/" for webhdfs
[
https://issues.apache.org/jira/browse/HDFS-14369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807306#comment-16807306
]
Akira Ajisaka commented on HDFS-14369:
--
Filed HADOOP-16226 to fix the problem.
> RBF: Fix trailing "/" for webhdfs
> -
>
> Key: HDFS-14369
> URL: https://issues.apache.org/jira/browse/HDFS-14369
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-14369-HDFS-13891-regressiontest-001.patch,
> HDFS-14369-HDFS-13891.001.patch, HDFS-14369-HDFS-13891.002.patch,
> HDFS-14369-HDFS-13891.003.patch, HDFS-14369-HDFS-13891.004.patch
>
>
> WebHDFS doesn't trim trailing slash causing discrepancy in operations.
> Example below
> --
> Using HDFS API, two directory are listed.
> {code}
> $ hdfs dfs -ls hdfs://:/tmp/
> Found 2 items
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp1
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp2
> {code}
> Using WebHDFS API, only one directory is listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp/?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16387,"group":"supergroup","length":0,"modificationTime":1552016766769,"owner":"hdfs","pathSuffix":"tmp1","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
> ]}}
> {code}
> The mount table is as follows:
> {code}
> $ hdfs dfsrouteradmin -ls /tmp
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /tmp ns1->/tmp aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp1 ns1->/tmp/tmp1aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp2 ns2->/tmp/tmp2aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> {code}
> Without trailing thrash, two directories are listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":1541753421917,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753421917,"owner":"hdfs","pathSuffix":"tmp1","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"},
> {"accessTime":1541753429812,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753429812,"owner":"hdfs","pathSuffix":"tmp2","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"}
> ]}}
> {code}
> [~ajisakaa] Thanks for reporting this, I borrowed the text from
> HDFS-13972
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14394) Add -std=c99 / -std=gnu99 to libhdfs compile flags
[
https://issues.apache.org/jira/browse/HDFS-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807296#comment-16807296
]
Eric Yang edited comment on HDFS-14394 at 4/1/19 11:58 PM:
---
[~stakiar] It seems a little strange that it failed the first time, then it
worked the second time. The last section of the maven output looks like this:
{code:java}
[exec]
/home/eyang/test/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfspp_mini_dfs.h:124:
Failure
[exec] Expected: (nullptr) != (clusterInfo), actual: 8-byte object <00-00
00-00 00-00 00-00> vs NULL
[exec] #
[exec] # A fatal error has been detected by the Java Runtime Environment:
[exec] #
[exec] # SIGSEGV (0xb) at pc=0x00783b7e, pid=28414,
tid=0x7efc91ac38c0
[exec] #
[exec] # JRE version: OpenJDK Runtime Environment (8.0_151-b12) (build
1.8.0_151-b12)
[exec] # Java VM: OpenJDK 64-Bit Server VM (25.151-b12 mixed mode
linux-amd64 compressed oops)
[exec] # Problematic frame:
[exec] nmdCreate: Builder#build error:
[exec] RuntimeException: Although a UNIX domain socket path is configured
as /tmp/native_mini_dfs.sock.28414.846930886, we cannot start a
localDataXceiverServer because libhadoop cannot be
loaded.java.lang.RuntimeException: Although a UNIX domain socket path is
configured as /tmp/native_mini_dfs.sock.28414.846930886, we cannot start a
localDataXceiverServer because libhadoop cannot be loaded.
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:1209)
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:1178)
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1433)
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:509)
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2827)
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2733)
[exec] at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1697)
[exec] at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:913)
[exec] at
org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:520)
[exec] at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:479)
[exec] # C [hdfs_ext_hdfspp_test_shim_static+0x383b7e]
[exec] #
[exec] # Failed to write core dump. Core dumps have been disabled. To
enable core dumping, try "ulimit -c unlimited" before starting Java again
[exec] #
[exec] # An error report file with more information is saved as:
[exec] #
/home/eyang/test/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfspp/tests/hs_err_pid28414.log
[exec] #
[exec] # If you would like to submit a bug report, please visit:
[exec] # http://bugreport.java.com/bugreport/crash.jsp
[exec] # The crash happened outside the Java Virtual Machine in native
code.
[exec] # See problematic frame for where to report the bug.
[exec] #
[exec]
[exec]
[exec] 85% tests passed, 6 tests failed out of 40
[exec]
[exec] Total Test time (real) = 96.60 sec
[exec]
[exec] The following tests FAILED:
[exec] 3 - test_test_libhdfs_zerocopy_hdfs_static (Failed)
[exec] 36 - test_hdfspp_mini_dfs_smoke_hdfspp_test_shim_static
(OTHER_FAULT)
[exec] 37 - libhdfs_mini_stress_valgrind_hdfspp_test_static (Failed)
[exec] 38 - memcheck_libhdfs_mini_stress_valgrind_hdfspp_test_static
(Failed)
[exec] 39 - test_libhdfs_mini_stress_hdfspp_test_shim_static (Failed)
[exec] 40 - test_hdfs_ext_hdfspp_test_shim_static (OTHER_FAULT)
[exec] Errors while running CTest{code}
It was not able to find libhadoop native library, even though I did ran with
mvn clean install -Pnative in hadoop-common-project follow by the same compile
command in hadoop-hdfs-native-client project.
There are some difference between
[https://stackoverflow.com/questions/10461331/what-are-the-incompatible-differences-between-c99-and-c11|c++11
and c99]. I don't know if this will create instability in libhdfspp to use
c99. Gcc 4.5+ can enforce c99 standard by passing -std=c99 -pedantic-errors
-fextended-identifiers instead of gnu99 to prevent gnu features to be included.
was (Author: eyang):
[~stakiar] It seems a little strange that it failed the first time, then it
worked the second time. The last section of the maven output looks like this:
{code:java}
[exec]
/home/eyang/test/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfspp_mini_dfs.h:124:
Failure
[exec] Expected: (nullptr) != (clusterInfo), actual: 8-byte object <00-00
[jira] [Commented] (HDFS-14369) RBF: Fix trailing "/" for webhdfs
[
https://issues.apache.org/jira/browse/HDFS-14369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807303#comment-16807303
]
Akira Ajisaka commented on HDFS-14369:
--
Agreed to fix this Path.normalizePath problem in a separate issue. Thank you
[~ayushtkn] for finding the root cause.
> RBF: Fix trailing "/" for webhdfs
> -
>
> Key: HDFS-14369
> URL: https://issues.apache.org/jira/browse/HDFS-14369
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-14369-HDFS-13891-regressiontest-001.patch,
> HDFS-14369-HDFS-13891.001.patch, HDFS-14369-HDFS-13891.002.patch,
> HDFS-14369-HDFS-13891.003.patch, HDFS-14369-HDFS-13891.004.patch
>
>
> WebHDFS doesn't trim trailing slash causing discrepancy in operations.
> Example below
> --
> Using HDFS API, two directory are listed.
> {code}
> $ hdfs dfs -ls hdfs://:/tmp/
> Found 2 items
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp1
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp2
> {code}
> Using WebHDFS API, only one directory is listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp/?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16387,"group":"supergroup","length":0,"modificationTime":1552016766769,"owner":"hdfs","pathSuffix":"tmp1","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
> ]}}
> {code}
> The mount table is as follows:
> {code}
> $ hdfs dfsrouteradmin -ls /tmp
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /tmp ns1->/tmp aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp1 ns1->/tmp/tmp1aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp2 ns2->/tmp/tmp2aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> {code}
> Without trailing thrash, two directories are listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":1541753421917,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753421917,"owner":"hdfs","pathSuffix":"tmp1","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"},
> {"accessTime":1541753429812,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753429812,"owner":"hdfs","pathSuffix":"tmp2","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"}
> ]}}
> {code}
> [~ajisakaa] Thanks for reporting this, I borrowed the text from
> HDFS-13972
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14394) Add -std=c99 / -std=gnu99 to libhdfs compile flags
[
https://issues.apache.org/jira/browse/HDFS-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807296#comment-16807296
]
Eric Yang commented on HDFS-14394:
--
[~stakiar] It seems a little strange that it failed the first time, then it
worked the second time. The last section of the maven output looks like this:
{code:java}
[exec]
/home/eyang/test/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfspp_mini_dfs.h:124:
Failure
[exec] Expected: (nullptr) != (clusterInfo), actual: 8-byte object <00-00
00-00 00-00 00-00> vs NULL
[exec] #
[exec] # A fatal error has been detected by the Java Runtime Environment:
[exec] #
[exec] # SIGSEGV (0xb) at pc=0x00783b7e, pid=28414,
tid=0x7efc91ac38c0
[exec] #
[exec] # JRE version: OpenJDK Runtime Environment (8.0_151-b12) (build
1.8.0_151-b12)
[exec] # Java VM: OpenJDK 64-Bit Server VM (25.151-b12 mixed mode
linux-amd64 compressed oops)
[exec] # Problematic frame:
[exec] nmdCreate: Builder#build error:
[exec] RuntimeException: Although a UNIX domain socket path is configured
as /tmp/native_mini_dfs.sock.28414.846930886, we cannot start a
localDataXceiverServer because libhadoop cannot be
loaded.java.lang.RuntimeException: Although a UNIX domain socket path is
configured as /tmp/native_mini_dfs.sock.28414.846930886, we cannot start a
localDataXceiverServer because libhadoop cannot be loaded.
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:1209)
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:1178)
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1433)
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:509)
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2827)
[exec] at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2733)
[exec] at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1697)
[exec] at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:913)
[exec] at
org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:520)
[exec] at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:479)
[exec] # C [hdfs_ext_hdfspp_test_shim_static+0x383b7e]
[exec] #
[exec] # Failed to write core dump. Core dumps have been disabled. To
enable core dumping, try "ulimit -c unlimited" before starting Java again
[exec] #
[exec] # An error report file with more information is saved as:
[exec] #
/home/eyang/test/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfspp/tests/hs_err_pid28414.log
[exec] #
[exec] # If you would like to submit a bug report, please visit:
[exec] # http://bugreport.java.com/bugreport/crash.jsp
[exec] # The crash happened outside the Java Virtual Machine in native
code.
[exec] # See problematic frame for where to report the bug.
[exec] #
[exec]
[exec]
[exec] 85% tests passed, 6 tests failed out of 40
[exec]
[exec] Total Test time (real) = 96.60 sec
[exec]
[exec] The following tests FAILED:
[exec] 3 - test_test_libhdfs_zerocopy_hdfs_static (Failed)
[exec] 36 - test_hdfspp_mini_dfs_smoke_hdfspp_test_shim_static
(OTHER_FAULT)
[exec] 37 - libhdfs_mini_stress_valgrind_hdfspp_test_static (Failed)
[exec] 38 - memcheck_libhdfs_mini_stress_valgrind_hdfspp_test_static
(Failed)
[exec] 39 - test_libhdfs_mini_stress_hdfspp_test_shim_static (Failed)
[exec] 40 - test_hdfs_ext_hdfspp_test_shim_static (OTHER_FAULT)
[exec] Errors while running CTest{code}
It was not able to find libhadoop native library, even though I did ran with
mvn clean install -Pnative in hadoop-common-project follow by the same compile
command in hadoop-hdfs-native-client project.
> Add -std=c99 / -std=gnu99 to libhdfs compile flags
> --
>
> Key: HDFS-14394
> URL: https://issues.apache.org/jira/browse/HDFS-14394
> Project: Hadoop HDFS
> Issue Type: Task
> Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HDFS-14394.001.patch
>
>
> libhdfs compilation currently does not enforce a minimum required C version.
> As of today, the libhdfs build on Hadoop QA works, but when built on a
> machine with an outdated gcc / cc version where C89 is the default,
> compilation fails due to errors such as:
> {code}
>
[jira] [Commented] (HDFS-13699) Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP
[
https://issues.apache.org/jira/browse/HDFS-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807295#comment-16807295
]
Konstantin Shvachko commented on HDFS-13699:
Chen, thanks for the comprehensive analysis of upgrade scenarios. Sounds like
we should not have unexpected failures during rolling upgrades.
??DFSClient code has many Perconditions checks??
We should stop using Preconditions as a substitute for java {{assert}}.
{{assert}} conditions are not even included in the binaries when compiled for
execution in production, while Preconditions are always executed. So if only
for performance reasons we should prefer asserts. Another reason is that
Preconditions is the major use case for Guava in Hadoop, and we could not
neither upgrade nor get rid of it for a long time. So it's an anti-pattern,
which we should avoid replicating.
And in tests we use junit Asserts.
# Check the style warnings, some of them can be fixed.
# Unused imports: DNConf, SaslDataTransferServer,
# Avoid *-imports like in TestMultipleNNPortQOP:
{code}
import static org.apache.hadoop.hdfs.DFSConfigKeys.*;
import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.*;
import static org.junit.Assert.*;
{code}
> Add DFSClient sending handshake token to DataNode, and allow DataNode
> overwrite downstream QOP
> --
>
> Key: HDFS-13699
> URL: https://issues.apache.org/jira/browse/HDFS-13699
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Attachments: HDFS-13699.001.patch, HDFS-13699.002.patch,
> HDFS-13699.003.patch, HDFS-13699.004.patch, HDFS-13699.005.patch,
> HDFS-13699.006.patch, HDFS-13699.007.patch, HDFS-13699.008.patch,
> HDFS-13699.WIP.001.patch
>
>
> Given the other Jiras under HDFS-13541, this Jira is to allow DFSClient to
> redirect the encrypt secret to DataNode. The encrypted message is the QOP
> that client and NameNode have used. DataNode decrypts the message and enforce
> the QOP for the client connection. Also, this Jira will also include
> overwriting downstream QOP, as mentioned in the HDFS-13541 design doc.
> Namely, this is to allow inter-DN QOP that is different from client-DN QOP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-3246) pRead equivalent for direct read path
[ https://issues.apache.org/jira/browse/HDFS-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HDFS-3246: --- Status: Patch Available (was: Open) > pRead equivalent for direct read path > - > > Key: HDFS-3246 > URL: https://issues.apache.org/jira/browse/HDFS-3246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, performance >Affects Versions: 3.0.0-alpha1 >Reporter: Henry Robinson >Assignee: Sahil Takiar >Priority: Major > Attachments: HDFS-3246.001.patch, HDFS-3246.002.patch, > HDFS-3246.003.patch, HDFS-3246.004.patch, HDFS-3246.005.patch, > HDFS-3246.006.patch, HDFS-3246.007.patch > > > There is no pread equivalent in ByteBufferReadable. We should consider adding > one. It would be relatively easy to implement for the distributed case > (certainly compared to HDFS-2834), since DFSInputStream does most of the > heavy lifting. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-3246) pRead equivalent for direct read path
[ https://issues.apache.org/jira/browse/HDFS-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HDFS-3246: --- Status: Open (was: Patch Available) > pRead equivalent for direct read path > - > > Key: HDFS-3246 > URL: https://issues.apache.org/jira/browse/HDFS-3246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, performance >Affects Versions: 3.0.0-alpha1 >Reporter: Henry Robinson >Assignee: Sahil Takiar >Priority: Major > Attachments: HDFS-3246.001.patch, HDFS-3246.002.patch, > HDFS-3246.003.patch, HDFS-3246.004.patch, HDFS-3246.005.patch, > HDFS-3246.006.patch, HDFS-3246.007.patch > > > There is no pread equivalent in ByteBufferReadable. We should consider adding > one. It would be relatively easy to implement for the distributed case > (certainly compared to HDFS-2834), since DFSInputStream does most of the > heavy lifting. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807289#comment-16807289
]
Arpit Agarwal commented on HDFS-10477:
--
Just a thought - it may be okay to skip this function altogether, and let the
block invalidations be handled by the full block reports. That will also
stagger the invalidation work.
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.004.patch, HDFS-10477.005.patch, HDFS-10477.006.patch,
> HDFS-10477.007.patch, HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
>
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807280#comment-16807280
]
Arpit Agarwal commented on HDFS-10477:
--
+1 the patch lgtm. Thanks for fixing this [~jojochuang].
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.004.patch, HDFS-10477.005.patch, HDFS-10477.006.patch,
> HDFS-10477.007.patch, HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 280219 over-replicated blocks on 10.142.27.14:1004 during recommissioning
>
[jira] [Commented] (HDFS-14394) Add -std=c99 / -std=gnu99 to libhdfs compile flags
[
https://issues.apache.org/jira/browse/HDFS-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807272#comment-16807272
]
Sahil Takiar commented on HDFS-14394:
-
[~eyang] what was the compilation error that you hit? I double checked and I'm
able to compile the patch locally.
Is there a specific reason we don't want the {{-std=c99}} flag in libhdfs++?
Would moving the C_FLAGS setting to {{src/main/native/libhdfs/CMakeLists.txt}}
fix this?
I'm not sure what the policy is on using {{-std=c99}} or {{-std=gnu99}},
although we were already using {{gnu99}} for Solaris builds. I don't think
there is a specific reason we need {{gnu99}} over {{c99}} though.
> Add -std=c99 / -std=gnu99 to libhdfs compile flags
> --
>
> Key: HDFS-14394
> URL: https://issues.apache.org/jira/browse/HDFS-14394
> Project: Hadoop HDFS
> Issue Type: Task
> Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HDFS-14394.001.patch
>
>
> libhdfs compilation currently does not enforce a minimum required C version.
> As of today, the libhdfs build on Hadoop QA works, but when built on a
> machine with an outdated gcc / cc version where C89 is the default,
> compilation fails due to errors such as:
> {code}
> /build/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jclasses.c:106:5:
> error: ‘for’ loop initial declarations are only allowed in C99 mode
> for (int i = 0; i < numCachedClasses; i++) {
> ^
> /build/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jclasses.c:106:5:
> note: use option -std=c99 or -std=gnu99 to compile your code
> {code}
> We should add the -std=c99 / -std=gnu99 flags to libhdfs compilation so that
> we can enforce C99 as the minimum required version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1330) Add a docker compose for Ozone deployment with Recon.
[ https://issues.apache.org/jira/browse/HDDS-1330?focusedWorklogId=221535=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221535 ] ASF GitHub Bot logged work on HDDS-1330: Author: ASF GitHub Bot Created on: 01/Apr/19 23:15 Start Date: 01/Apr/19 23:15 Worklog Time Spent: 10m Work Description: vivekratnavel commented on issue #669: HDDS-1330 : Add a docker compose for Ozone deployment with Recon. URL: https://github.com/apache/hadoop/pull/669#issuecomment-478780526 +1 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 221535) Time Spent: 0.5h (was: 20m) > Add a docker compose for Ozone deployment with Recon. > - > > Key: HDDS-1330 > URL: https://issues.apache.org/jira/browse/HDDS-1330 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: HDDS-1330-000.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > * Add a docker compose for Ozone deployment with Recon. > * Test out Recon container key service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1355) Only FQDN is accepted for OM rpc address in secure environment
[
https://issues.apache.org/jira/browse/HDDS-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar reassigned HDDS-1355:
Assignee: Ajay Kumar
> Only FQDN is accepted for OM rpc address in secure environment
> --
>
> Key: HDDS-1355
> URL: https://issues.apache.org/jira/browse/HDDS-1355
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Ajay Kumar
>Priority: Major
>
> While the scm address can be a host name (relative to the current search
> domain) if the om address is just a hostname and not a FQDN a NPE is thrown:
> {code}
> 10 │ OZONE-SITE.XML_ozone.om.address: "om-0.om"
> 11 │ OZONE-SITE.XML_ozone.scm.client.address: "scm-0.scm"
> 12 │ OZONE-SITE.XML_ozone.scm.names: "scm-0.scm"
> {code}
> {code}
> 2019-03-29 14:37:52 ERROR OzoneManager:865 - Failed to start the OzoneManager.
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.om.OzoneManager.getSCMSignedCert(OzoneManager.java:1372)
> at
> org.apache.hadoop.ozone.om.OzoneManager.initializeSecurity(OzoneManager.java:1018)
> at org.apache.hadoop.ozone.om.OzoneManager.omInit(OzoneManager.java:971)
> at org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:928)
> at org.apache.hadoop.ozone.om.OzoneManager.main(OzoneManager.java:859)
> {code}
> I don't know what is the right validation rule here, but I am pretty sure
> that NPE should be avoided and a meaningful error should be thrown. (and the
> behaviour should be the same for scm and om)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14394) Add -std=c99 / -std=gnu99 to libhdfs compile flags
[
https://issues.apache.org/jira/browse/HDFS-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807253#comment-16807253
]
Hadoop QA commented on HDFS-14394:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
44m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
13s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14394 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964488/HDFS-14394.001.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 74742028e1a7 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ab2bda5 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26563/testReport/ |
| Max. process+thread count | 1464 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26563/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add -std=c99 / -std=gnu99 to libhdfs compile flags
> --
>
> Key: HDFS-14394
> URL: https://issues.apache.org/jira/browse/HDFS-14394
> Project: Hadoop HDFS
> Issue Type: Task
> Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HDFS-14394.001.patch
>
>
> libhdfs compilation currently does not enforce a minimum required C version.
> As of today, the libhdfs build on Hadoop QA works, but when built on a
> machine with an outdated gcc / cc version where C89 is the default,
> compilation fails due to errors such as:
> {code}
>
[jira] [Commented] (HDFS-14397) Backport HADOOP-15684 to branch-2
[
https://issues.apache.org/jira/browse/HDFS-14397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807248#comment-16807248
]
Hadoop QA commented on HDFS-14397:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} branch-2 passed with JDK v1.8.0_191 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} branch-2 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
54s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in branch-2 has 1
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} branch-2 passed with JDK v1.8.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed with JDK v1.8.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 47 unchanged - 1 fixed = 47 total (was 48) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK v1.8.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | hadoop.hdfs.TestTrashWithSecureEncryptionZones |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:da67579 |
| JIRA Issue | HDFS-14397 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964484/HDFS-14397-branch-2.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux
[jira] [Commented] (HDFS-14394) Add -std=c99 / -std=gnu99 to libhdfs compile flags
[
https://issues.apache.org/jira/browse/HDFS-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807241#comment-16807241
]
Eric Yang commented on HDFS-14394:
--
[~stakiar]
If a small workaround can be applied without project refactoring effort, I
think most people would prefer the work around.
In hadoop-hdfs-native-client project, there are c++ projects. The globally
defined C_FLAGS will change all sub-projects to use gnu99 standard including
libhdfspp project. Difference between c99 and gnu99 are gnu extensions that get
compiled into the binary. This may subject the produced artifact to work with
GPL license. I am not a lawyer, but I think this is not allowed in Apache
projects.
I suggested to refactor the project setup with cmake-maven-plugins for two
improvements.
# User does not need to have cmake install in the local system. It will get
downloaded as part of maven binary.
# making sure that distinct C_FLAGS is not shared between projects of C or C++
languages.
Patch 001 didn't compile hadoop-hdfs-native-client libhdfspp sub-project for
me.
> Add -std=c99 / -std=gnu99 to libhdfs compile flags
> --
>
> Key: HDFS-14394
> URL: https://issues.apache.org/jira/browse/HDFS-14394
> Project: Hadoop HDFS
> Issue Type: Task
> Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HDFS-14394.001.patch
>
>
> libhdfs compilation currently does not enforce a minimum required C version.
> As of today, the libhdfs build on Hadoop QA works, but when built on a
> machine with an outdated gcc / cc version where C89 is the default,
> compilation fails due to errors such as:
> {code}
> /build/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jclasses.c:106:5:
> error: ‘for’ loop initial declarations are only allowed in C99 mode
> for (int i = 0; i < numCachedClasses; i++) {
> ^
> /build/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jclasses.c:106:5:
> note: use option -std=c99 or -std=gnu99 to compile your code
> {code}
> We should add the -std=c99 / -std=gnu99 flags to libhdfs compilation so that
> we can enforce C99 as the minimum required version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14400) Namenode ExpiredHeartbeats metric
[
https://issues.apache.org/jira/browse/HDFS-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Palanisamy updated HDFS-14400:
--
Attachment: HDFS-14400-003.patch
> Namenode ExpiredHeartbeats metric
> -
>
> Key: HDFS-14400
> URL: https://issues.apache.org/jira/browse/HDFS-14400
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.2
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Minor
> Attachments: HDFS-14400-001.patch, HDFS-14400-002.patch,
> HDFS-14400-003.patch
>
>
> Noticed incorrect value in ExpiredHeartbeats metrics under namenode JMX.
> We will increment ExpiredHeartbeats count when Datanode is dead but somehow
> we missed to decrement when datanode is alive back.
> {code}
> { "name" : "Hadoop:service=NameNode,name=FSNamesystem", "modelerType" :
> "FSNamesystem", "tag.Context" : "dfs", "tag.TotalSyncTimes" : "7 ",
> "tag.HAState" : "active", ... "ExpiredHeartbeats" : 2, ... }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807229#comment-16807229
]
Hadoop QA commented on HDFS-14390:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 25m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
32s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 55s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
47s{color} | {color:green} hadoop-fs2img in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
59s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}245m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14390 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964467/HDFS-14390.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 2558aaa94ea0 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16
[jira] [Commented] (HDFS-14400) Namenode ExpiredHeartbeats metric
[
https://issues.apache.org/jira/browse/HDFS-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807224#comment-16807224
]
Hadoop QA commented on HDFS-14400:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 56s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 47 unchanged - 0 fixed = 48 total (was 47) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 51s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}128m 25s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
59s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}191m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14400 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964468/HDFS-14400-002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 23d92db918bf 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / da7f8c2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26560/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26560/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807219#comment-16807219
]
Wei-Chiu Chuang commented on HDFS-10477:
Test failures do not reproduce locally.
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.004.patch, HDFS-10477.005.patch, HDFS-10477.006.patch,
> HDFS-10477.007.patch, HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 280219 over-replicated blocks on 10.142.27.14:1004 during recommissioning
> 2016-05-26
[jira] [Commented] (HDFS-14394) Add -std=c99 / -std=gnu99 to libhdfs compile flags
[
https://issues.apache.org/jira/browse/HDFS-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807214#comment-16807214
]
Sahil Takiar commented on HDFS-14394:
-
Thanks for the input [~jojochuang], [~tlipcon]. Posted a patch that adds
{{set(CMAKE_C_FLAGS "-std=gnu99 ${CMAKE_C_FLAGS}")}} to {{HadoopCommon.cmake}}.
It looks like we already do this for Solaris builds.
[~eyang] not entirely sure if I follow your reasoning for splitting
{{hadoop-hdfs-native-client}} into several sub-projects, could you expand a bit
more? Are you referring to
[this|https://github.com/cmake-maven-project/cmake-maven-project] Maven plugin?
It certainly looks interesting. However, both of these changes look like
relatively large projects that are probably out of the scope of what this JIRA
is trying to address.
> Add -std=c99 / -std=gnu99 to libhdfs compile flags
> --
>
> Key: HDFS-14394
> URL: https://issues.apache.org/jira/browse/HDFS-14394
> Project: Hadoop HDFS
> Issue Type: Task
> Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HDFS-14394.001.patch
>
>
> libhdfs compilation currently does not enforce a minimum required C version.
> As of today, the libhdfs build on Hadoop QA works, but when built on a
> machine with an outdated gcc / cc version where C89 is the default,
> compilation fails due to errors such as:
> {code}
> /build/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jclasses.c:106:5:
> error: ‘for’ loop initial declarations are only allowed in C99 mode
> for (int i = 0; i < numCachedClasses; i++) {
> ^
> /build/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jclasses.c:106:5:
> note: use option -std=c99 or -std=gnu99 to compile your code
> {code}
> We should add the -std=c99 / -std=gnu99 flags to libhdfs compilation so that
> we can enforce C99 as the minimum required version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14394) Add -std=c99 / -std=gnu99 to libhdfs compile flags
[
https://issues.apache.org/jira/browse/HDFS-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HDFS-14394:
Attachment: HDFS-14394.001.patch
> Add -std=c99 / -std=gnu99 to libhdfs compile flags
> --
>
> Key: HDFS-14394
> URL: https://issues.apache.org/jira/browse/HDFS-14394
> Project: Hadoop HDFS
> Issue Type: Task
> Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HDFS-14394.001.patch
>
>
> libhdfs compilation currently does not enforce a minimum required C version.
> As of today, the libhdfs build on Hadoop QA works, but when built on a
> machine with an outdated gcc / cc version where C89 is the default,
> compilation fails due to errors such as:
> {code}
> /build/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jclasses.c:106:5:
> error: ‘for’ loop initial declarations are only allowed in C99 mode
> for (int i = 0; i < numCachedClasses; i++) {
> ^
> /build/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jclasses.c:106:5:
> note: use option -std=c99 or -std=gnu99 to compile your code
> {code}
> We should add the -std=c99 / -std=gnu99 flags to libhdfs compilation so that
> we can enforce C99 as the minimum required version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14394) Add -std=c99 / -std=gnu99 to libhdfs compile flags
[
https://issues.apache.org/jira/browse/HDFS-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HDFS-14394:
Status: Patch Available (was: Open)
> Add -std=c99 / -std=gnu99 to libhdfs compile flags
> --
>
> Key: HDFS-14394
> URL: https://issues.apache.org/jira/browse/HDFS-14394
> Project: Hadoop HDFS
> Issue Type: Task
> Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HDFS-14394.001.patch
>
>
> libhdfs compilation currently does not enforce a minimum required C version.
> As of today, the libhdfs build on Hadoop QA works, but when built on a
> machine with an outdated gcc / cc version where C89 is the default,
> compilation fails due to errors such as:
> {code}
> /build/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jclasses.c:106:5:
> error: ‘for’ loop initial declarations are only allowed in C99 mode
> for (int i = 0; i < numCachedClasses; i++) {
> ^
> /build/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jclasses.c:106:5:
> note: use option -std=c99 or -std=gnu99 to compile your code
> {code}
> We should add the -std=c99 / -std=gnu99 flags to libhdfs compilation so that
> we can enforce C99 as the minimum required version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14400) Namenode ExpiredHeartbeats metric
[
https://issues.apache.org/jira/browse/HDFS-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807193#comment-16807193
]
Hadoop QA commented on HDFS-14400:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 44s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}132m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14400 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964471/HDFS-14400-002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9dfef85834fe 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / da7f8c2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26561/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26561/testReport/ |
| Max. process+thread count | 5544 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Updated] (HDFS-14397) Backport HADOOP-15684 to branch-2
[ https://issues.apache.org/jira/browse/HDFS-14397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HDFS-14397: Attachment: HDFS-14397-branch-2.001.patch > Backport HADOOP-15684 to branch-2 > - > > Key: HDFS-14397 > URL: https://issues.apache.org/jira/browse/HDFS-14397 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Attachments: HDFS-14397-branch-2.000.patch, > HDFS-14397-branch-2.001.patch > > > As multi-SBN feature is already backported to branch-2, this is a follow-up > to backport HADOOP-15684. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1324) TestOzoneManagerHA seems to be flaky
[
https://issues.apache.org/jira/browse/HDDS-1324?focusedWorklogId=221476=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221476
]
ASF GitHub Bot logged work on HDDS-1324:
Author: ASF GitHub Bot
Created on: 01/Apr/19 20:58
Start Date: 01/Apr/19 20:58
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on issue #676: HDDS-1324.
TestOzoneManagerHA tests are flaky
URL: https://github.com/apache/hadoop/pull/676#issuecomment-478745426
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|::|--:|:|:|
| 0 | reexec | 22 | Docker mode activated. |
||| _ Prechecks _ |
| +1 | @author | 0 | The patch does not contain any @author tags. |
| +1 | test4tests | 0 | The patch appears to include 1 new or modified test
files. |
||| _ trunk Compile Tests _ |
| +1 | mvninstall | 1060 | trunk passed |
| +1 | compile | 61 | trunk passed |
| +1 | checkstyle | 22 | trunk passed |
| +1 | mvnsite | 34 | trunk passed |
| +1 | shadedclient | 710 | branch has no errors when building and testing
our client artifacts. |
| 0 | findbugs | 0 | Skipped patched modules with no Java source:
hadoop-ozone/integration-test |
| +1 | findbugs | 0 | trunk passed |
| +1 | javadoc | 22 | trunk passed |
||| _ Patch Compile Tests _ |
| +1 | mvninstall | 36 | the patch passed |
| +1 | compile | 25 | the patch passed |
| +1 | javac | 25 | the patch passed |
| +1 | checkstyle | 16 | the patch passed |
| +1 | mvnsite | 28 | the patch passed |
| +1 | whitespace | 0 | The patch has no whitespace issues. |
| +1 | shadedclient | 741 | patch has no errors when building and testing
our client artifacts. |
| 0 | findbugs | 0 | Skipped patched modules with no Java source:
hadoop-ozone/integration-test |
| +1 | findbugs | 0 | the patch passed |
| +1 | javadoc | 15 | the patch passed |
||| _ Other Tests _ |
| -1 | unit | 785 | integration-test in the patch failed. |
| +1 | asflicense | 30 | The patch does not generate ASF License warnings. |
| | | 3685 | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.ozone.om.TestOzoneManagerHA |
| | hadoop.ozone.om.TestScmChillMode |
| | hadoop.ozone.client.rpc.TestOzoneRpcClient |
| | hadoop.ozone.TestMiniChaosOzoneCluster |
| | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-676/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/676 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 22f1968a89dc 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / da7f8c2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| unit |
https://builds.apache.org/job/hadoop-multibranch/job/PR-676/1/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
|
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-676/1/testReport/ |
| Max. process+thread count | 3791 (vs. ulimit of 5500) |
| modules | C: hadoop-ozone/integration-test U:
hadoop-ozone/integration-test |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-676/1/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 221476)
Time Spent: 20m (was: 10m)
> TestOzoneManagerHA seems to be flaky
>
>
> Key: HDDS-1324
> URL: https://issues.apache.org/jira/browse/HDDS-1324
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Affects Versions: 0.5.0
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> TestOzoneManagerHA failed once with the following error:
> {code}
> [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
>
[jira] [Commented] (HDDS-1367) Add ability in Recon to track the growth rate of the cluster.
[ https://issues.apache.org/jira/browse/HDDS-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807174#comment-16807174 ] Dinesh Chitlangia commented on HDDS-1367: - [~avijayan] Thanks for filing this jira. This will definitely be very useful for end users and admins. Do you intend to answer the growth question from a storage only standpoint or also for number of read/write ops ? > Add ability in Recon to track the growth rate of the cluster. > -- > > Key: HDDS-1367 > URL: https://issues.apache.org/jira/browse/HDDS-1367 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Priority: Major > Fix For: 0.5.0 > > > Recon should be able to answer the question "How fast is the cluster growing, > by week, by month, by day?", which gives the user an idea of the usage stats > of the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1211) Test SCMChillMode failing randomly in Jenkins run
[
https://issues.apache.org/jira/browse/HDDS-1211?focusedWorklogId=221462=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221462
]
ASF GitHub Bot logged work on HDDS-1211:
Author: ASF GitHub Bot
Created on: 01/Apr/19 20:26
Start Date: 01/Apr/19 20:26
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #543: HDDS-1211. Test
SCMChillMode failing randomly in Jenkins run
URL: https://github.com/apache/hadoop/pull/543#discussion_r270536031
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestScmChillMode.java
##
@@ -225,36 +210,36 @@ public void testIsScmInChillModeAndForceExit() throws
Exception {
}
- @Test(timeout=300_000)
+ @Test(timeout = 300_000)
public void testSCMChillMode() throws Exception {
-MiniOzoneCluster.Builder clusterBuilder = MiniOzoneCluster.newBuilder(conf)
-.setHbInterval(1000)
-.setNumDatanodes(3)
Review comment:
Looks like the test was previously using 3 DNs and now it will use just 1
DN. Correct?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 221462)
Time Spent: 1h 50m (was: 1h 40m)
> Test SCMChillMode failing randomly in Jenkins run
> -
>
> Key: HDDS-1211
> URL: https://issues.apache.org/jira/browse/HDDS-1211
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available, pushed-to-craterlake
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:389) at
> org.apache.hadoop.ozone.om.TestScmChillMode.testSCMChillMode(TestScmChillMode.java:286)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1211) Test SCMChillMode failing randomly in Jenkins run
[
https://issues.apache.org/jira/browse/HDDS-1211?focusedWorklogId=221461=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221461
]
ASF GitHub Bot logged work on HDDS-1211:
Author: ASF GitHub Bot
Created on: 01/Apr/19 20:26
Start Date: 01/Apr/19 20:26
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #543: HDDS-1211. Test
SCMChillMode failing randomly in Jenkins run
URL: https://github.com/apache/hadoop/pull/543#discussion_r270535305
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestScmChillMode.java
##
@@ -158,25 +154,21 @@ public void testChillModeOperations() throws Exception {
cluster.stop();
Review comment:
Let's move the `cluster.stop()` call to an `@After` method. Actually it
looks like you already have cluster.shutdown(). Why do we need a stop call here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 221461)
Time Spent: 1h 40m (was: 1.5h)
> Test SCMChillMode failing randomly in Jenkins run
> -
>
> Key: HDDS-1211
> URL: https://issues.apache.org/jira/browse/HDDS-1211
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available, pushed-to-craterlake
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:389) at
> org.apache.hadoop.ozone.om.TestScmChillMode.testSCMChillMode(TestScmChillMode.java:286)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-534) ozone jmxget does not work
[
https://issues.apache.org/jira/browse/HDDS-534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shweta reassigned HDDS-534:
---
Assignee: Shweta
> ozone jmxget does not work
> --
>
> Key: HDDS-534
> URL: https://issues.apache.org/jira/browse/HDDS-534
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Namit Maheshwari
>Assignee: Shweta
>Priority: Major
>
> {code:java}
> [root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone jmxget
> ERROR: jmxget is not COMMAND nor fully qualified CLASSNAME.
> Usage: ozone [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
> OPTIONS is none or any of:
> --buildpaths attempt to add class files from build tree
> --config dir Hadoop config directory
> --daemon (start|status|stop) operate on a daemon
> --debug turn on shell script debug mode
> --help usage information
> --hostnames list[,of,host,names] hosts to use in worker mode
> --hosts filename list of hosts to use in worker mode
> --loglevel level set the log4j level for this command
> --workers turn on worker mode
> SUBCOMMAND is one of:
> Admin Commands:
> jmxget get JMX exported values from NameNode or DataNode.
> Client Commands:
> classpath prints the class path needed to get the hadoop jar and the required
> libraries
> envvars display computed Hadoop environment variables
> freon runs an ozone data generator
> fs run a filesystem command on Ozone file system. Equivalent to 'hadoop fs'
> genconf generate minimally required ozone configs and output to
> ozone-site.xml in specified path
> genesis runs a collection of ozone benchmarks to help with tuning.
> getozoneconf get ozone config values from configuration
> noz ozone debug tool, convert ozone metadata into relational data
> scmcli run the CLI of the Storage Container Manager
> sh command line interface for object store operations
> version print the version
> Daemon Commands:
> datanode run a HDDS datanode
> om Ozone Manager
> scm run the Storage Container Manager service
> SUBCOMMAND may print help when invoked w/o parameters or with -h.
> {code}
> As in the logs above jmxget is an option, but does not work.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13875) EOFException log spam in Datanode
[
https://issues.apache.org/jira/browse/HDFS-13875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807145#comment-16807145
]
Karthik Palanisamy commented on HDFS-13875:
---
[~ajayydv]
I log very generic message when DataInputStream is empty. Would you like to
change or handle any?
{code}
LOG.debug("Failed to read SASL message from client at " +
peer.getRemoteAddressString() + ". Likely client has closed the socket", e);
{code}
Currently, SaslExceptions are throwable in DataXceiver and these are logged as
error in the log.
My only concern is log spam.
> EOFException log spam in Datanode
> -
>
> Key: HDFS-13875
> URL: https://issues.apache.org/jira/browse/HDFS-13875
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Minor
> Attachments: HDFS-13875.001.patch, HDFS-13875.002.patch
>
>
> Ambari checks datanode liveness by simply connecting to data transfer port.
> But this connection will be closed after a successful TCP handshake without
> any data transfer. Due to which datanode encountered EOFExcetion when reading
> an encrypted message from the closed socket.
> This issue addressed in HDFS-9572. But not handled for encrypted data
> transfer(SASL message).
> {code:java}
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:361)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:180)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:112)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:194)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1324) TestOzoneManagerHA seems to be flaky
[
https://issues.apache.org/jira/browse/HDDS-1324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-1324:
-
Labels: pull-request-available (was: )
> TestOzoneManagerHA seems to be flaky
>
>
> Key: HDDS-1324
> URL: https://issues.apache.org/jira/browse/HDDS-1324
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Affects Versions: 0.5.0
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: pull-request-available
>
> TestOzoneManagerHA failed once with the following error:
> {code}
> [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 105.931 s <<< FAILURE! - in org.apache.hadoop.ozone.om.TestOzoneManagerHA
> [ERROR] testOMRetryProxy(org.apache.hadoop.ozone.om.TestOzoneManagerHA) Time
> elapsed: 21.781 s <<< FAILURE!
> java.lang.AssertionError: expected:<30> but was:<10>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.testOMRetryProxy(TestOzoneManagerHA.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1324) TestOzoneManagerHA seems to be flaky
[
https://issues.apache.org/jira/browse/HDDS-1324?focusedWorklogId=221446=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221446
]
ASF GitHub Bot logged work on HDDS-1324:
Author: ASF GitHub Bot
Created on: 01/Apr/19 19:56
Start Date: 01/Apr/19 19:56
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #676:
HDDS-1324. TestOzoneManagerHA tests are flaky
URL: https://github.com/apache/hadoop/pull/676
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 221446)
Time Spent: 10m
Remaining Estimate: 0h
> TestOzoneManagerHA seems to be flaky
>
>
> Key: HDDS-1324
> URL: https://issues.apache.org/jira/browse/HDDS-1324
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Affects Versions: 0.5.0
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> TestOzoneManagerHA failed once with the following error:
> {code}
> [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 105.931 s <<< FAILURE! - in org.apache.hadoop.ozone.om.TestOzoneManagerHA
> [ERROR] testOMRetryProxy(org.apache.hadoop.ozone.om.TestOzoneManagerHA) Time
> elapsed: 21.781 s <<< FAILURE!
> java.lang.AssertionError: expected:<30> but was:<10>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.testOMRetryProxy(TestOzoneManagerHA.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14399) Backport HDFS-10536 to branch-2
[ https://issues.apache.org/jira/browse/HDFS-14399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-14399: -- Resolution: Fixed Fix Version/s: 2.10.0 Status: Resolved (was: Patch Available) > Backport HDFS-10536 to branch-2 > --- > > Key: HDFS-14399 > URL: https://issues.apache.org/jira/browse/HDFS-14399 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Critical > Fix For: 2.10.0 > > Attachments: HDFS-14399-branch-2.000.patch > > > As multi-SBN feature is already backported to branch-2, this is a follow-up > to backport HDFS-10536. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14399) Backport HDFS-10536 to branch-2
[ https://issues.apache.org/jira/browse/HDFS-14399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807122#comment-16807122 ] Chen Liang commented on HDFS-14399: --- The failed tests ran successfully in my local run. +1 to v000 patch. I've committed to branch-2, thanks [~csun] for the contribution! > Backport HDFS-10536 to branch-2 > --- > > Key: HDFS-14399 > URL: https://issues.apache.org/jira/browse/HDFS-14399 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Critical > Attachments: HDFS-14399-branch-2.000.patch > > > As multi-SBN feature is already backported to branch-2, this is a follow-up > to backport HDFS-10536. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807109#comment-16807109
]
Hadoop QA commented on HDFS-10477:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 38s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 36s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-10477 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964465/HDFS-10477.007.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 31c93d06a7f5 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 856cbf6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26558/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26558/testReport/ |
| Max. process+thread count | 4776 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Created] (HDDS-1367) Add ability in Recon to track the growth rate of the cluster.
Aravindan Vijayan created HDDS-1367: --- Summary: Add ability in Recon to track the growth rate of the cluster. Key: HDDS-1367 URL: https://issues.apache.org/jira/browse/HDDS-1367 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Recon Reporter: Aravindan Vijayan Fix For: 0.5.0 Recon should be able to answer the question "How fast is the cluster growing, by week, by month, by day?", which gives the user an idea of the usage stats of the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1366) Add ability in Recon to track the number of small files in an Ozone cluster.
Aravindan Vijayan created HDDS-1366: --- Summary: Add ability in Recon to track the number of small files in an Ozone cluster. Key: HDDS-1366 URL: https://issues.apache.org/jira/browse/HDDS-1366 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Recon Reporter: Aravindan Vijayan Assignee: sarun singla Fix For: 0.5.0 Ozone users may want to track the number of small files they have in their cluster and where they are present. Recon can help them with the information by iterating the OM Key Table and dividing the keys into different buckets based on the data size. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1329) Update documentation for Ozone-0.4.0 release
[ https://issues.apache.org/jira/browse/HDDS-1329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807079#comment-16807079 ] Anu Engineer commented on HDDS-1329: # Architecture page for Security – A high level overview of how Ozone security works. # Setting up a secure cluster – Pages on how to setup a secure cluster. # Command Line tools - the new CLIs if we have any # General documentation update. This does not block the RC0 of Ozone release and voting, We will get this done this week. > Update documentation for Ozone-0.4.0 release > > > Key: HDDS-1329 > URL: https://issues.apache.org/jira/browse/HDDS-1329 > Project: Hadoop Distributed Data Store > Issue Type: Task >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Blocker > > We need to update documenation of Ozone for all the new features which is > part of 0.4.0 release. This is a 0.4.0 blocker JIRA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14400) Namenode ExpiredHeartbeats metric
[
https://issues.apache.org/jira/browse/HDFS-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Palanisamy updated HDFS-14400:
--
Attachment: HDFS-14400-002.patch
> Namenode ExpiredHeartbeats metric
> -
>
> Key: HDFS-14400
> URL: https://issues.apache.org/jira/browse/HDFS-14400
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.2
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Minor
> Attachments: HDFS-14400-001.patch, HDFS-14400-002.patch
>
>
> Noticed incorrect value in ExpiredHeartbeats metrics under namenode JMX.
> We will increment ExpiredHeartbeats count when Datanode is dead but somehow
> we missed to decrement when datanode is alive back.
> {code}
> { "name" : "Hadoop:service=NameNode,name=FSNamesystem", "modelerType" :
> "FSNamesystem", "tag.Context" : "dfs", "tag.TotalSyncTimes" : "7 ",
> "tag.HAState" : "active", ... "ExpiredHeartbeats" : 2, ... }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1337) Handle GroupMismatchException in OzoneClient
[
https://issues.apache.org/jira/browse/HDDS-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807072#comment-16807072
]
Hadoop QA commented on HDDS-1337:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 7m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
7s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
31s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange} hadoop-ozone: The patch generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 33s{color}
| {color:red} hadoop-hdds in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 38s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.container.common.TestDatanodeStateMachine |
| | hadoop.ozone.client.rpc.TestContainerStateMachineFailures |
| | hadoop.ozone.TestStorageContainerManager |
| | hadoop.ozone.scm.pipeline.TestPipelineManagerMXBean |
| | hadoop.ozone.client.rpc.TestContainerStateMachine |
\\
\\
|| Subsystem || Report/Notes
[jira] [Updated] (HDFS-14400) Namenode ExpiredHeartbeats metric
[
https://issues.apache.org/jira/browse/HDFS-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Palanisamy updated HDFS-14400:
--
Attachment: (was: HDFS-14400-002.patch)
> Namenode ExpiredHeartbeats metric
> -
>
> Key: HDFS-14400
> URL: https://issues.apache.org/jira/browse/HDFS-14400
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.2
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Minor
> Attachments: HDFS-14400-001.patch
>
>
> Noticed incorrect value in ExpiredHeartbeats metrics under namenode JMX.
> We will increment ExpiredHeartbeats count when Datanode is dead but somehow
> we missed to decrement when datanode is alive back.
> {code}
> { "name" : "Hadoop:service=NameNode,name=FSNamesystem", "modelerType" :
> "FSNamesystem", "tag.Context" : "dfs", "tag.TotalSyncTimes" : "7 ",
> "tag.HAState" : "active", ... "ExpiredHeartbeats" : 2, ... }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14400) Namenode ExpiredHeartbeats metric
[
https://issues.apache.org/jira/browse/HDFS-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807066#comment-16807066
]
Karthik Palanisamy commented on HDFS-14400:
---
[~goiri] Yes, ExpiredHeartbeats is just a counter for dead datanode.
ExpiredHeartbeats is not used by any other functions, it only exposed for
metrics.
{quote}The problem would be if we double counted.
{quote}
It will not be double counted. Assume, One datanode is marked as dead. We
incremented expiredHeartbeat counter to 1, When same datanode node is alive
again after sometime, the counter is not decremented to 0.
The fix is to handle this scenario.
> Namenode ExpiredHeartbeats metric
> -
>
> Key: HDFS-14400
> URL: https://issues.apache.org/jira/browse/HDFS-14400
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.2
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Minor
> Attachments: HDFS-14400-001.patch, HDFS-14400-002.patch
>
>
> Noticed incorrect value in ExpiredHeartbeats metrics under namenode JMX.
> We will increment ExpiredHeartbeats count when Datanode is dead but somehow
> we missed to decrement when datanode is alive back.
> {code}
> { "name" : "Hadoop:service=NameNode,name=FSNamesystem", "modelerType" :
> "FSNamesystem", "tag.Context" : "dfs", "tag.TotalSyncTimes" : "7 ",
> "tag.HAState" : "active", ... "ExpiredHeartbeats" : 2, ... }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1333) OzoneFileSystem can't work with spark/hadoop2.7 because incompatible security classes
[ https://issues.apache.org/jira/browse/HDDS-1333?focusedWorklogId=221405=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221405 ] ASF GitHub Bot logged work on HDDS-1333: Author: ASF GitHub Bot Created on: 01/Apr/19 18:33 Start Date: 01/Apr/19 18:33 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #653: HDDS-1333. OzoneFileSystem can't work with spark/hadoop2.7 because incompatible security classes URL: https://github.com/apache/hadoop/pull/653#issuecomment-478693613 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 29 | Docker mode activated. | ||| _ Prechecks _ | | 0 | yamllint | 0 | yamllint was not available. | | +1 | @author | 1 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ ozone-0.4 Compile Tests _ | | 0 | mvndep | 36 | Maven dependency ordering for branch | | +1 | mvninstall | 1283 | ozone-0.4 passed | | +1 | compile | 1185 | ozone-0.4 passed | | +1 | checkstyle | 231 | ozone-0.4 passed | | -1 | mvnsite | 66 | common in ozone-0.4 failed. | | -1 | mvnsite | 46 | ozonefs in ozone-0.4 failed. | | +1 | shadedclient | 796 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-hdds/docs hadoop-ozone/dist | | -1 | findbugs | 38 | common in ozone-0.4 failed. | | -1 | findbugs | 35 | ozonefs in ozone-0.4 failed. | | +1 | javadoc | 145 | ozone-0.4 passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 25 | Maven dependency ordering for patch | | -1 | mvninstall | 28 | common in the patch failed. | | -1 | mvninstall | 24 | dist in the patch failed. | | -1 | mvninstall | 26 | ozonefs in the patch failed. | | +1 | compile | 956 | the patch passed | | +1 | javac | 956 | the patch passed | | +1 | checkstyle | 212 | the patch passed | | -1 | mvnsite | 45 | common in the patch failed. | | -1 | mvnsite | 39 | ozonefs in the patch failed. | | +1 | shellcheck | 0 | There were no new shellcheck issues. | | +1 | shelldocs | 31 | There were no new shelldocs issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 749 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-hdds/docs hadoop-ozone/dist | | -1 | findbugs | 38 | common in the patch failed. | | -1 | findbugs | 37 | ozonefs in the patch failed. | | +1 | javadoc | 144 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 33 | docs in the patch passed. | | -1 | unit | 43 | common in the patch failed. | | +1 | unit | 36 | dist in the patch passed. | | -1 | unit | 37 | ozonefs in the patch failed. | | +1 | asflicense | 48 | The patch does not generate ASF License warnings. | | | | 6955 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-653/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/653 | | Optional Tests | dupname asflicense mvnsite compile javac javadoc mvninstall unit shadedclient findbugs checkstyle shellcheck shelldocs xml yamllint | | uname | Linux c40fcfbeb157 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct 31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | ozone-0.4 / fb7844d | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-653/3/artifact/out/branch-mvnsite-hadoop-ozone_common.txt | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-653/3/artifact/out/branch-mvnsite-hadoop-ozone_ozonefs.txt | | shellcheck | v0.4.6 | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-653/3/artifact/out/branch-findbugs-hadoop-ozone_common.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-653/3/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-653/3/artifact/out/patch-mvninstall-hadoop-ozone_common.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-653/3/artifact/out/patch-mvninstall-hadoop-ozone_dist.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-653/3/artifact/out/patch-mvninstall-hadoop-ozone_ozonefs.txt
[jira] [Updated] (HDFS-14400) Namenode ExpiredHeartbeats metric
[
https://issues.apache.org/jira/browse/HDFS-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Palanisamy updated HDFS-14400:
--
Attachment: HDFS-14400-002.patch
> Namenode ExpiredHeartbeats metric
> -
>
> Key: HDFS-14400
> URL: https://issues.apache.org/jira/browse/HDFS-14400
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.2
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Minor
> Attachments: HDFS-14400-001.patch, HDFS-14400-002.patch
>
>
> Noticed incorrect value in ExpiredHeartbeats metrics under namenode JMX.
> We will increment ExpiredHeartbeats count when Datanode is dead but somehow
> we missed to decrement when datanode is alive back.
> {code}
> { "name" : "Hadoop:service=NameNode,name=FSNamesystem", "modelerType" :
> "FSNamesystem", "tag.Context" : "dfs", "tag.TotalSyncTimes" : "7 ",
> "tag.HAState" : "active", ... "ExpiredHeartbeats" : 2, ... }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1355) Only FQDN is accepted for OM rpc address in secure environment
[
https://issues.apache.org/jira/browse/HDDS-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807062#comment-16807062
]
Ajay Kumar commented on HDDS-1355:
--
[~elek] thanks for reporting this. I tried to reproduce this with our secure
docker but it doesn't fail with this error. Is it specific to kubernetes. Om
host name in our ozonesecure is om (i.e not fqdn) and it works.
> Only FQDN is accepted for OM rpc address in secure environment
> --
>
> Key: HDDS-1355
> URL: https://issues.apache.org/jira/browse/HDDS-1355
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Priority: Major
>
> While the scm address can be a host name (relative to the current search
> domain) if the om address is just a hostname and not a FQDN a NPE is thrown:
> {code}
> 10 │ OZONE-SITE.XML_ozone.om.address: "om-0.om"
> 11 │ OZONE-SITE.XML_ozone.scm.client.address: "scm-0.scm"
> 12 │ OZONE-SITE.XML_ozone.scm.names: "scm-0.scm"
> {code}
> {code}
> 2019-03-29 14:37:52 ERROR OzoneManager:865 - Failed to start the OzoneManager.
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.om.OzoneManager.getSCMSignedCert(OzoneManager.java:1372)
> at
> org.apache.hadoop.ozone.om.OzoneManager.initializeSecurity(OzoneManager.java:1018)
> at org.apache.hadoop.ozone.om.OzoneManager.omInit(OzoneManager.java:971)
> at org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:928)
> at org.apache.hadoop.ozone.om.OzoneManager.main(OzoneManager.java:859)
> {code}
> I don't know what is the right validation rule here, but I am pretty sure
> that NPE should be avoided and a meaningful error should be thrown. (and the
> behaviour should be the same for scm and om)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1354) Kerberos principal configuration of OzoneManager doesn't use FQDN
[
https://issues.apache.org/jira/browse/HDDS-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807056#comment-16807056
]
Ajay Kumar commented on HDDS-1354:
--
[~elek] thanks for reporting this. OM behaivour is also same. Rpc Server is
binded to socket address returned by {{OmUtils.getOmAddress}}.
> Kerberos principal configuration of OzoneManager doesn't use FQDN
> -
>
> Key: HDDS-1354
> URL: https://issues.apache.org/jira/browse/HDDS-1354
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Elek, Marton
>Assignee: Ajay Kumar
>Priority: Major
>
> In the "*.kerberos.principal" settings hadoop supports the _HOST variable
> which is replaced to the fully qualified domain name.
> For example:
> {code}
> OZONE-SITE.XML_hdds.scm.kerberos.principal: "scm/_h...@example.com"
> {code}
> It works well with scm but for om it uses the hostname instead of the FQDN.
> (SCM uses the HddsServerUtil.getScmBlockClientBindAddress which uses the
> _bind_ address but the om uses the om rpc address).
> I would suggest to use the same behaviour for both SCM and OM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1354) Kerberos principal configuration of OzoneManager doesn't use FQDN
[
https://issues.apache.org/jira/browse/HDDS-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807056#comment-16807056
]
Ajay Kumar edited comment on HDDS-1354 at 4/1/19 6:20 PM:
--
[~elek] thanks for reporting this. OM behavior is also same. Rpc Server is
binded to socket address returned by {{OmUtils.getOmAddress}}.
was (Author: ajayydv):
[~elek] thanks for reporting this. OM behaivour is also same. Rpc Server is
binded to socket address returned by {{OmUtils.getOmAddress}}.
> Kerberos principal configuration of OzoneManager doesn't use FQDN
> -
>
> Key: HDDS-1354
> URL: https://issues.apache.org/jira/browse/HDDS-1354
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Elek, Marton
>Assignee: Ajay Kumar
>Priority: Major
>
> In the "*.kerberos.principal" settings hadoop supports the _HOST variable
> which is replaced to the fully qualified domain name.
> For example:
> {code}
> OZONE-SITE.XML_hdds.scm.kerberos.principal: "scm/_h...@example.com"
> {code}
> It works well with scm but for om it uses the hostname instead of the FQDN.
> (SCM uses the HddsServerUtil.getScmBlockClientBindAddress which uses the
> _bind_ address but the om uses the om rpc address).
> I would suggest to use the same behaviour for both SCM and OM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1347) In OM HA getS3Secret call Should happen only leader OM
[ https://issues.apache.org/jira/browse/HDDS-1347?focusedWorklogId=221392=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221392 ] ASF GitHub Bot logged work on HDDS-1347: Author: ASF GitHub Bot Created on: 01/Apr/19 18:09 Start Date: 01/Apr/19 18:09 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #670: HDDS-1347. In OM HA getS3Secret call Should happen only leader OM. URL: https://github.com/apache/hadoop/pull/670#issuecomment-478685029 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 40 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 68 | Maven dependency ordering for branch | | +1 | mvninstall | 1174 | trunk passed | | +1 | compile | 110 | trunk passed | | +1 | checkstyle | 31 | trunk passed | | +1 | mvnsite | 69 | trunk passed | | +1 | shadedclient | 785 | branch has no errors when building and testing our client artifacts. | | +1 | findbugs | 114 | trunk passed | | +1 | javadoc | 64 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 12 | Maven dependency ordering for patch | | +1 | mvninstall | 76 | the patch passed | | +1 | compile | 96 | the patch passed | | +1 | cc | 96 | the patch passed | | +1 | javac | 96 | the patch passed | | -0 | checkstyle | 23 | hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 58 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 762 | patch has no errors when building and testing our client artifacts. | | +1 | findbugs | 116 | the patch passed | | +1 | javadoc | 55 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 35 | common in the patch passed. | | +1 | unit | 52 | ozone-manager in the patch passed. | | +1 | asflicense | 28 | The patch does not generate ASF License warnings. | | | | 3832 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-670/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/670 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 9da069f35dd1 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 04f1db8 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-670/2/artifact/out/diff-checkstyle-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-670/2/testReport/ | | Max. process+thread count | 392 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/ozone-manager U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-670/2/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 221392) Time Spent: 0.5h (was: 20m) > In OM HA getS3Secret call Should happen only leader OM > -- > > Key: HDDS-1347 > URL: https://issues.apache.org/jira/browse/HDDS-1347 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > In Om HA getS3Secret should happen only leader OM. > > > The reason is similar to initiateMultipartUpload. For more info refer > HDDS-1319 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For
[jira] [Work logged] (HDDS-1358) Recon Server REST API not working as expected.
[ https://issues.apache.org/jira/browse/HDDS-1358?focusedWorklogId=221386=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221386 ] ASF GitHub Bot logged work on HDDS-1358: Author: ASF GitHub Bot Created on: 01/Apr/19 18:04 Start Date: 01/Apr/19 18:04 Worklog Time Spent: 10m Work Description: vivekratnavel commented on issue #668: HDDS-1358 : Recon Server REST API not working as expected. URL: https://github.com/apache/hadoop/pull/668#issuecomment-478683382 +1 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 221386) Time Spent: 40m (was: 0.5h) > Recon Server REST API not working as expected. > -- > > Key: HDDS-1358 > URL: https://issues.apache.org/jira/browse/HDDS-1358 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Critical > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: HDDS-1358-000.patch, HDDS-1358-001.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Guice Jetty integration that is being used for Recon Server API layer is not > working as expected. Fixing that in this JIRA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashvin updated HDFS-14390:
--
Attachment: HDFS-14390.003.patch
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch,
> HDFS-14390.003.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDDS-1353) Metrics scm_pipeline_metrics_num_pipeline_creation_failed keeps increasing because of BackgroundPipelineCreator
[
https://issues.apache.org/jira/browse/HDDS-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDDS-1353 started by Aravindan Vijayan.
---
> Metrics scm_pipeline_metrics_num_pipeline_creation_failed keeps increasing
> because of BackgroundPipelineCreator
> ---
>
> Key: HDDS-1353
> URL: https://issues.apache.org/jira/browse/HDDS-1353
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Elek, Marton
>Assignee: Aravindan Vijayan
>Priority: Minor
> Labels: newbie
>
> There is a {{BackgroundPipelineCreator}} thread in SCM which runs in a fixed
> interval and tries to create pipelines. This BackgroundPipelineCreator uses
> {{IOException}} as exit criteria (no more pipelines can be created). In each
> run of BackgroundPipelineCreator we exit when we are not able to create any
> more pipelines, i.e. when we get IOException while trying to create the
> pipeline. This means that
> {{scm_pipeline_metrics_num_pipeline_creation_failed}} value will get
> incremented in each run of BackgroundPipelineCreator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1260) Create Recon Server lifecyle integration with Ozone.
[ https://issues.apache.org/jira/browse/HDDS-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807008#comment-16807008 ] Hudson commented on HDDS-1260: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16317 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16317/]) HDDS-1260. Create Recon Server lifecyle integration with Ozone. (#643) (bharat: rev 04f1db89366c8edfe9bba683f29d62dcc6600c2b) * (edit) hadoop-ozone/ozone-recon/pom.xml * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServer.java * (add) hadoop-ozone/ozone-recon/src/main/resources/webapps/recon/index.html * (edit) hadoop-ozone/common/src/main/bin/ozone * (edit) hadoop-ozone/dist/pom.xml > Create Recon Server lifecyle integration with Ozone. > > > Key: HDDS-1260 > URL: https://issues.apache.org/jira/browse/HDDS-1260 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Vivek Ratnavel Subramanian >Priority: Critical > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > * Create the lifecycle scripts (start/stop) for Recon Server along with Shell > interface like the other components. > * Verify configurations are being picked up by Recon Server on startup. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Moved] (HDFS-14403) Cost-Based RPC FairCallQueue
[ https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen moved HADOOP-16224 to HDFS-14403: - Component/s: (was: ipc) namenode ipc Key: HDFS-14403 (was: HADOOP-16224) Project: Hadoop HDFS (was: Hadoop Common) > Cost-Based RPC FairCallQueue > > > Key: HDFS-14403 > URL: https://issues.apache.org/jira/browse/HDFS-14403 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ipc, namenode >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Labels: qos, rpc > > HADOOP-15016 initially described extensions to the Hadoop FairCallQueue > encompassing both cost-based analysis of incoming RPCs, as well as support > for reservations of RPC capacity for system/platform users. This JIRA intends > to track the former, as HADOOP-15016 was repurposed to more specifically > focus on the reservation portion of the work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14403) Cost-Based RPC FairCallQueue
[ https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807006#comment-16807006 ] Erik Krogen commented on HDFS-14403: I moved this to the HDFS project as I remembered that this has HDFS-specific components. Fully specced out design document, and initial patch, coming in the next few days. > Cost-Based RPC FairCallQueue > > > Key: HDFS-14403 > URL: https://issues.apache.org/jira/browse/HDFS-14403 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ipc, namenode >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Labels: qos, rpc > > HADOOP-15016 initially described extensions to the Hadoop FairCallQueue > encompassing both cost-based analysis of incoming RPCs, as well as support > for reservations of RPC capacity for system/platform users. This JIRA intends > to track the former, as HADOOP-15016 was repurposed to more specifically > focus on the reservation portion of the work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-10477:
---
Attachment: HDFS-10477.007.patch
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.004.patch, HDFS-10477.005.patch, HDFS-10477.006.patch,
> HDFS-10477.007.patch, HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 280219 over-replicated blocks on 10.142.27.14:1004 during recommissioning
> 2016-05-26 20:13:25,370 INFO
>
[jira] [Updated] (HDDS-1356) Wrong response code in s3g in case of an invalid access key
[
https://issues.apache.org/jira/browse/HDDS-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDDS-1356:
---
Target Version/s: 0.5.0 (was: 0.4.0)
> Wrong response code in s3g in case of an invalid access key
> ---
>
> Key: HDDS-1356
> URL: https://issues.apache.org/jira/browse/HDDS-1356
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Priority: Major
>
> In case of a wrong aws credential the s3g returns with HTTP 500:
> {code}
> [hadoop@om-0 keytabs]$ aws s3api --endpoint=http://s3g-0.s3g:9878
> create-bucket --bucket qwe
> An error occurred (500) when calling the CreateBucket operation (reached max
> retries: 4): Internal Server Error
> {code}
> And throws an exception server side:
> {code}
> s3g-0 s3g 3ff4582bec94fee02ae4babcd4294c5a1c46cf7a6f750bfd5de4e894e41663c5,
> signature=73ea5e939f47de1389e26624c91444d6b88fa70c64e5ee1e39e6804269736a99,
> awsAccessKeyId=scm/om-0.om.perf.svc.cluster.lo...@example.co
> s3g-0 s3g at
> org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511)
> s3g-0 s3g at org.apache.hadoop.ipc.Client.call(Client.java:1457)
> s3g-0 s3g at org.apache.hadoop.ipc.Client.call(Client.java:1367)
> s3g-0 s3g at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> s3g-0 s3g at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> s3g-0 s3g at com.sun.proxy.$Proxy77.submitRequest(Unknown Source)
> s3g-0 s3g at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown
> Source)
> s3g-0 s3g at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> s3g-0 s3g at java.lang.reflect.Method.invoke(Method.java:498)
> s3g-0 s3g at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> s3g-0 s3g at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> s3g-0 s3g at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> s3g-0 s3g at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> s3g-0 s3g at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> s3g-0 s3g at com.sun.proxy.$Proxy77.submitRequest(Unknown Source)
> s3g-0 s3g at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> s3g-0 s3g at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> s3g-0 s3g at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> s3g-0 s3g at java.lang.reflect.Method.invoke(Method.java:498)
> s3g-0 s3g at
> org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66)
> s3g-0 s3g at com.sun.proxy.$Proxy77.submitRequest(Unknown Source)
> s3g-0 s3g at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:284)
> s3g-0 s3g at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceList(OzoneManagerProtocolClientSideTranslatorPB.java:1097)
> s3g-0 s3g at
> org.apache.hadoop.ozone.client.rpc.RpcClient.getScmAddressForClient(RpcClient.java:219)
> s3g-0 s3g at
> org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:148)
> s3g-0 s3g at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> s3g-0 s3g at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> s3g-0 s3g at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> s3g-0 s3g at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> s3g-0 s3g at
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:291)
> s3g-0 s3g at
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:92)
> s3g-0 s3g at
> org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:108)
> s3g-0 s3g at
> org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:68)
> s3g-0 s3g at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> s3g-0 s3g at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> s3g-0 s3g at
>
[jira] [Comment Edited] (HDDS-1337) Handle GroupMismatchException in OzoneClient
[
https://issues.apache.org/jira/browse/HDDS-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806964#comment-16806964
]
Shashikant Banerjee edited comment on HDDS-1337 at 4/1/19 5:10 PM:
---
Thanks [~linyiqun] for the review.
{code:java}
+System.out.println("Pipeline exist " + raftServer.getPipelineIds()
+.contains(pipeline.getId()));
+/* GenericTestUtils.waitFor(
+() -> (!raftServer.getPipelineIds()
+.contains(pipeline.getId())), 500, 100 * 1000 );*/
{code}
The logging line has been removed as its not really required. The wait
condition just ensures the pipeline has been removed on the datanode.
Rest of the review comments and checkstyle issues have been addressed with
patch v4.
was (Author: shashikant):
Thanks [~linyiqun] for the review.
{code:java}
+System.out.println("Pipeline exist " + raftServer.getPipelineIds()
+.contains(pipeline.getId()));
+/* GenericTestUtils.waitFor(
+() -> (!raftServer.getPipelineIds()
+.contains(pipeline.getId())), 500, 100 * 1000 );*/
{code}
The logging line has been removed as its not really required. The wait
condition just ensures the pipeline has been removed on the datanode.
Rest of the review . comments and checkstyle issues have been addressed with
patch v4.
> Handle GroupMismatchException in OzoneClient
>
>
> Key: HDDS-1337
> URL: https://issues.apache.org/jira/browse/HDDS-1337
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Client
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Blocker
> Labels: Blocker
> Fix For: 0.4.0
>
> Attachments: HDDS-1337.000.patch, HDDS-1337.001.patch,
> HDDS-1337.002.patch, HDDS-1337.004.patch
>
>
> If a pipeline gets destroyed in ozone client, ozone client may hit
> GroupMismatchException from Ratis. In cases as such, client should exclude
> the pipeline and retry write to a different block.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1337) Handle GroupMismatchException in OzoneClient
[ https://issues.apache.org/jira/browse/HDDS-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-1337: -- Summary: Handle GroupMismatchException in OzoneClient (was: HandleGroupMismatchException in OzoneClient) > Handle GroupMismatchException in OzoneClient > > > Key: HDDS-1337 > URL: https://issues.apache.org/jira/browse/HDDS-1337 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Blocker > Labels: Blocker > Fix For: 0.4.0 > > Attachments: HDDS-1337.000.patch, HDDS-1337.001.patch, > HDDS-1337.002.patch, HDDS-1337.004.patch > > > If a pipeline gets destroyed in ozone client, ozone client may hit > GroupMismatchException from Ratis. In cases as such, client should exclude > the pipeline and retry write to a different block. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1353) Metrics scm_pipeline_metrics_num_pipeline_creation_failed keeps increasing because of BackgroundPipelineCreator
[
https://issues.apache.org/jira/browse/HDDS-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806988#comment-16806988
]
Aravindan Vijayan commented on HDDS-1353:
-
[~elek] I can take a stab at this.
> Metrics scm_pipeline_metrics_num_pipeline_creation_failed keeps increasing
> because of BackgroundPipelineCreator
> ---
>
> Key: HDDS-1353
> URL: https://issues.apache.org/jira/browse/HDDS-1353
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Elek, Marton
>Priority: Minor
> Labels: newbie
>
> There is a {{BackgroundPipelineCreator}} thread in SCM which runs in a fixed
> interval and tries to create pipelines. This BackgroundPipelineCreator uses
> {{IOException}} as exit criteria (no more pipelines can be created). In each
> run of BackgroundPipelineCreator we exit when we are not able to create any
> more pipelines, i.e. when we get IOException while trying to create the
> pipeline. This means that
> {{scm_pipeline_metrics_num_pipeline_creation_failed}} value will get
> incremented in each run of BackgroundPipelineCreator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1353) Metrics scm_pipeline_metrics_num_pipeline_creation_failed keeps increasing because of BackgroundPipelineCreator
[
https://issues.apache.org/jira/browse/HDDS-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan reassigned HDDS-1353:
---
Assignee: Aravindan Vijayan
> Metrics scm_pipeline_metrics_num_pipeline_creation_failed keeps increasing
> because of BackgroundPipelineCreator
> ---
>
> Key: HDDS-1353
> URL: https://issues.apache.org/jira/browse/HDDS-1353
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Elek, Marton
>Assignee: Aravindan Vijayan
>Priority: Minor
> Labels: newbie
>
> There is a {{BackgroundPipelineCreator}} thread in SCM which runs in a fixed
> interval and tries to create pipelines. This BackgroundPipelineCreator uses
> {{IOException}} as exit criteria (no more pipelines can be created). In each
> run of BackgroundPipelineCreator we exit when we are not able to create any
> more pipelines, i.e. when we get IOException while trying to create the
> pipeline. This means that
> {{scm_pipeline_metrics_num_pipeline_creation_failed}} value will get
> incremented in each run of BackgroundPipelineCreator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1260) Create Recon Server lifecyle integration with Ozone.
[ https://issues.apache.org/jira/browse/HDDS-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1260: - Resolution: Fixed Fix Version/s: 0.5.0 Status: Resolved (was: Patch Available) Thank You [~vivekratnavel] for the contribution and [~avijayan] for review. I have committed this to trunk. > Create Recon Server lifecyle integration with Ozone. > > > Key: HDDS-1260 > URL: https://issues.apache.org/jira/browse/HDDS-1260 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Vivek Ratnavel Subramanian >Priority: Critical > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > * Create the lifecycle scripts (start/stop) for Recon Server along with Shell > interface like the other components. > * Verify configurations are being picked up by Recon Server on startup. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1260) Create Recon Server lifecyle integration with Ozone.
[ https://issues.apache.org/jira/browse/HDDS-1260?focusedWorklogId=221355=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221355 ] ASF GitHub Bot logged work on HDDS-1260: Author: ASF GitHub Bot Created on: 01/Apr/19 16:59 Start Date: 01/Apr/19 16:59 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #643: HDDS-1260. Create Recon Server lifecycle integration with Ozone. URL: https://github.com/apache/hadoop/pull/643 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 221355) Time Spent: 2.5h (was: 2h 20m) > Create Recon Server lifecyle integration with Ozone. > > > Key: HDDS-1260 > URL: https://issues.apache.org/jira/browse/HDDS-1260 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Vivek Ratnavel Subramanian >Priority: Critical > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > * Create the lifecycle scripts (start/stop) for Recon Server along with Shell > interface like the other components. > * Verify configurations are being picked up by Recon Server on startup. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1322) Hugo errors when building Ozone
[
https://issues.apache.org/jira/browse/HDDS-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806978#comment-16806978
]
Doroszlai, Attila commented on HDDS-1322:
-
Thanks [~arpitagarwal] for the review/commit.
> Hugo errors when building Ozone
> ---
>
> Key: HDDS-1322
> URL: https://issues.apache.org/jira/browse/HDDS-1322
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Arpit Agarwal
>Assignee: Doroszlai, Attila
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.0
>
> Attachments: HDDS-1322.001.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I see some odd hugo errors when building Ozone, even though I am not building
> docs.
> {code}
> $ mvn -B -q clean compile install -DskipTests=true -Dmaven.javadoc.skip=true
> -Dmaven.site.skip=true -DskipShade -Phdds
> Error: unknown command "0.4.0-SNAPSHOT" for "hugo"
> Run 'hugo --help' for usage.
> .../hadoop-hdds/docs/target
> Error: unknown command "0.4.0-SNAPSHOT" for "hugo"
> Run 'hugo --help' for usage.
> .../hadoop-hdds/docs/target
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1164) Add New blockade Tests to test Replica Manager
[
https://issues.apache.org/jira/browse/HDDS-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806977#comment-16806977
]
Shashikant Banerjee commented on HDDS-1164:
---
Thanks [~nilotpalnandi] for updating the patch. The patch overall looks good to
me.
{code:java}
+ In this test, one of the datanodes (first datanode) cannot communicate with
+ other two datanodes.
+ Two datanodes (first datanode and second datanode),
+ on different network paritions, cannot communicate with SCM.
+ Expectation :
+ The container replica state in first datanode should be open.
+ The container replica states can be either 'closed'
+ in both second and third datanode, or,
+ 'open' in second datanode and 'quasi-closed' in third datanode.
{code}
If a set of two datanodes cannot communicate to SCM as well as the other
datanode, the containers on these two datanodes cannot go to CLOSED or quasi
closed state on their own. Can you please check this?
> Add New blockade Tests to test Replica Manager
> --
>
> Key: HDDS-1164
> URL: https://issues.apache.org/jira/browse/HDDS-1164
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Nilotpal Nandi
>Assignee: Nilotpal Nandi
>Priority: Major
> Labels: postpone-to-craterlake
> Attachments: HDDS-1164.001.patch, HDDS-1164.002.patch,
> HDDS-1164.003.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1337) HandleGroupMismatchException in OzoneClient
[ https://issues.apache.org/jira/browse/HDDS-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-1337: -- Attachment: HDDS-1337.004.patch > HandleGroupMismatchException in OzoneClient > --- > > Key: HDDS-1337 > URL: https://issues.apache.org/jira/browse/HDDS-1337 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Blocker > Labels: Blocker > Fix For: 0.4.0 > > Attachments: HDDS-1337.000.patch, HDDS-1337.001.patch, > HDDS-1337.002.patch, HDDS-1337.004.patch > > > If a pipeline gets destroyed in ozone client, ozone client may hit > GroupMismatchException from Ratis. In cases as such, client should exclude > the pipeline and retry write to a different block. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1337) HandleGroupMismatchException in OzoneClient
[ https://issues.apache.org/jira/browse/HDDS-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-1337: -- Attachment: (was: HDDS-1337.004.patch) > HandleGroupMismatchException in OzoneClient > --- > > Key: HDDS-1337 > URL: https://issues.apache.org/jira/browse/HDDS-1337 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Blocker > Labels: Blocker > Fix For: 0.4.0 > > Attachments: HDDS-1337.000.patch, HDDS-1337.001.patch, > HDDS-1337.002.patch > > > If a pipeline gets destroyed in ozone client, ozone client may hit > GroupMismatchException from Ratis. In cases as such, client should exclude > the pipeline and retry write to a different block. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1322) Hugo errors when building Ozone
[
https://issues.apache.org/jira/browse/HDDS-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806969#comment-16806969
]
Hudson commented on HDDS-1322:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16316 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16316/])
HDDS-1322. Hugo errors when building Ozone (#671) (arp7: rev
36756703f062d944daa148a5709dc24506106e4f)
* (edit) hadoop-hdds/docs/pom.xml
> Hugo errors when building Ozone
> ---
>
> Key: HDDS-1322
> URL: https://issues.apache.org/jira/browse/HDDS-1322
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Arpit Agarwal
>Assignee: Doroszlai, Attila
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.0
>
> Attachments: HDDS-1322.001.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I see some odd hugo errors when building Ozone, even though I am not building
> docs.
> {code}
> $ mvn -B -q clean compile install -DskipTests=true -Dmaven.javadoc.skip=true
> -Dmaven.site.skip=true -DskipShade -Phdds
> Error: unknown command "0.4.0-SNAPSHOT" for "hugo"
> Run 'hugo --help' for usage.
> .../hadoop-hdds/docs/target
> Error: unknown command "0.4.0-SNAPSHOT" for "hugo"
> Run 'hugo --help' for usage.
> .../hadoop-hdds/docs/target
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1255) Refactor ozone acceptance test to allow run in secure mode
[ https://issues.apache.org/jira/browse/HDDS-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806968#comment-16806968 ] Hudson commented on HDDS-1255: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16316 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16316/]) HDDS-1255. Refactor ozone acceptance test to allow run in secure mode (elek: rev 5f951ea2e39ae4dfe554942baeec05849cd7d3c2) * (delete) hadoop-ozone/dist/src/main/smoketest/security/ozone-secure.robot * (edit) hadoop-ozone/dist/src/main/smoketest/commonlib.robot * (edit) hadoop-ozone/dist/src/main/smoketest/test.sh * (add) hadoop-ozone/dist/src/main/smoketest/security/ozone-secure-fs.robot * (edit) hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config * (add) hadoop-ozone/dist/src/main/smoketest/security/ozone-secure-s3.robot * (edit) hadoop-ozone/dist/src/main/smoketest/s3/commonawslib.robot * (add) hadoop-ozone/dist/src/main/smoketest/__init__.robot * (edit) hadoop-ozone/dist/src/main/smoketest/auditparser/auditparser.robot > Refactor ozone acceptance test to allow run in secure mode > -- > > Key: HDDS-1255 > URL: https://issues.apache.org/jira/browse/HDDS-1255 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Time Spent: 6.5h > Remaining Estimate: 0h > > Refactor ozone acceptance test to allow run in secure mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1337) HandleGroupMismatchException in OzoneClient
[ https://issues.apache.org/jira/browse/HDDS-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-1337: -- Attachment: (was: HDDS-1337.003.patch) > HandleGroupMismatchException in OzoneClient > --- > > Key: HDDS-1337 > URL: https://issues.apache.org/jira/browse/HDDS-1337 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Blocker > Labels: Blocker > Fix For: 0.4.0 > > Attachments: HDDS-1337.000.patch, HDDS-1337.001.patch, > HDDS-1337.002.patch, HDDS-1337.004.patch > > > If a pipeline gets destroyed in ozone client, ozone client may hit > GroupMismatchException from Ratis. In cases as such, client should exclude > the pipeline and retry write to a different block. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1337) HandleGroupMismatchException in OzoneClient
[
https://issues.apache.org/jira/browse/HDDS-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806964#comment-16806964
]
Shashikant Banerjee commented on HDDS-1337:
---
Thanks [~linyiqun] for the review.
{code:java}
+System.out.println("Pipeline exist " + raftServer.getPipelineIds()
+.contains(pipeline.getId()));
+/* GenericTestUtils.waitFor(
+() -> (!raftServer.getPipelineIds()
+.contains(pipeline.getId())), 500, 100 * 1000 );*/
{code}
The logging line has been removed as its not really required. The wait
condition just ensures the pipeline has been removed on the datanode.
Rest of the review . comments and checkstyle issues have been addressed with
patch v4.
> HandleGroupMismatchException in OzoneClient
> ---
>
> Key: HDDS-1337
> URL: https://issues.apache.org/jira/browse/HDDS-1337
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Client
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Blocker
> Labels: Blocker
> Fix For: 0.4.0
>
> Attachments: HDDS-1337.000.patch, HDDS-1337.001.patch,
> HDDS-1337.002.patch, HDDS-1337.003.patch, HDDS-1337.004.patch
>
>
> If a pipeline gets destroyed in ozone client, ozone client may hit
> GroupMismatchException from Ratis. In cases as such, client should exclude
> the pipeline and retry write to a different block.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1337) HandleGroupMismatchException in OzoneClient
[ https://issues.apache.org/jira/browse/HDDS-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-1337: -- Attachment: HDDS-1337.004.patch > HandleGroupMismatchException in OzoneClient > --- > > Key: HDDS-1337 > URL: https://issues.apache.org/jira/browse/HDDS-1337 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Blocker > Labels: Blocker > Fix For: 0.4.0 > > Attachments: HDDS-1337.000.patch, HDDS-1337.001.patch, > HDDS-1337.002.patch, HDDS-1337.003.patch, HDDS-1337.004.patch > > > If a pipeline gets destroyed in ozone client, ozone client may hit > GroupMismatchException from Ratis. In cases as such, client should exclude > the pipeline and retry write to a different block. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1365) Fix error handling in KeyValueContainerCheck
Supratim Deka created HDDS-1365: --- Summary: Fix error handling in KeyValueContainerCheck Key: HDDS-1365 URL: https://issues.apache.org/jira/browse/HDDS-1365 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Datanode Reporter: Supratim Deka Assignee: Supratim Deka Error handling and propagation in KeyValueContainerCheck needs to be based on throwing IOException instead of passing an error code to the calling function. HDDS-1163 implemented the basic framework using a mix of error code return and exception handling. There is added complexity because exceptions deep inside the call chain are being caught and translated to error code return values. The goal is to change all error handling in this class to use Exceptions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1189) Recon Aggregate DB schema and ORM
[ https://issues.apache.org/jira/browse/HDDS-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806950#comment-16806950 ] Elek, Marton commented on HDDS-1189: Thanks you the patch [~swagle]. This patch introduces a lot of new dependencies. I think the LICENSE file should be updated. Jooq seems to be apache 2 but HSQLDB seems to have a different licence ( and I didn't check the transitive dependencies of spring-jdbc). (Ping me offline if you need help. I think the easiest way to fix this is to compare the distribution folder with and without this patch and check the licence of all the new jar files. All the non Apache should be added to the LICENCE. ) > Recon Aggregate DB schema and ORM > - > > Key: HDDS-1189 > URL: https://issues.apache.org/jira/browse/HDDS-1189 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Affects Versions: 0.5.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle >Priority: Major > Fix For: 0.5.0 > > Attachments: HDDS-1189.01.patch, HDDS-1189.02.patch, > HDDS-1189.03.patch, HDDS-1189.04.patch > > > _Objectives_ > - Define V1 of the db schema for recon service > - The current proposal is to use jOOQ as the ORM for SQL interaction. For two > main reasons: a) powerful DSL for querying, that abstracts out SQL dialects, > b) Allows code to schema and schema to code seamless transition, critical for > creating DDL through the code and unit testing across versions of the > application. > - Add e2e unit tests suite for Recon entities, created based on the design doc -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1337) HandleGroupMismatchException in OzoneClient
[
https://issues.apache.org/jira/browse/HDDS-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806949#comment-16806949
]
Hadoop QA commented on HDDS-1337:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
40s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
0s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange} hadoop-ozone: The patch generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 52s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 33s{color}
| {color:red} hadoop-hdds in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 26s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.container.common.TestDatanodeStateMachine |
| | hadoop.ozone.client.rpc.TestContainerStateMachine |
| | hadoop.ozone.client.rpc.TestContainerStateMachineFailures |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
[jira] [Resolved] (HDDS-1229) Concurrency issues with Background Block Delete
[ https://issues.apache.org/jira/browse/HDDS-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Supratim Deka resolved HDDS-1229. - Resolution: Not A Problem This is not an issue because HDDS-1163 adopted a simple approach to resolving the concurrency with background block delete. Every time a chunk lookup fails when checking the block DB, the checker retries lookup for the specific block. If the block is not found in DB, it means background delete has removed it and the missing chunk is not actually a corruption. > Concurrency issues with Background Block Delete > --- > > Key: HDDS-1229 > URL: https://issues.apache.org/jira/browse/HDDS-1229 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Datanode >Reporter: Supratim Deka >Priority: Major > > HDDS-1163 takes a simplistic approach to deal with concurrent block deletes > on a container, > when the metadata scanner is checking existence of chunks for each block in > the Container Block DB. > As part of HDDS-1663 checkBlockDB() just does a retry if any inconsistency is > detected during a concurrency window. The retry is expected to succeed > because the new DB iterator will not include any of the blocks being > processed by the concurrent background delete. If retry fails, then the > inconsistency is ignored expecting the next iteration of the metadata scanner > will avoid running concurrently with the same container. > This Jira is raised to explore a more predictable (yet simple) mechanism to > deal with this concurrency. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1333) OzoneFileSystem can't work with spark/hadoop2.7 because incompatible security classes
[
https://issues.apache.org/jira/browse/HDDS-1333?focusedWorklogId=221339=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221339
]
ASF GitHub Bot logged work on HDDS-1333:
Author: ASF GitHub Bot
Created on: 01/Apr/19 15:56
Start Date: 01/Apr/19 15:56
Worklog Time Spent: 10m
Work Description: elek commented on pull request #653: HDDS-1333.
OzoneFileSystem can't work with spark/hadoop2.7 because incompatible security
classes
URL: https://github.com/apache/hadoop/pull/653#discussion_r270937426
##
File path: hadoop-ozone/dist/dev-support/bin/dist-layout-stitching
##
@@ -114,6 +114,7 @@ run cp
"${ROOT}/hadoop-ozone/objectstore-service/target/hadoop-ozone-objectstore
cp -r "${ROOT}/hadoop-hdds/docs/target/classes/docs" ./
#Copy docker compose files
-run cp -p -R "${ROOT}/hadoop-ozone/dist/src/main/compose" .
+#compose files are preprocessed: properties (eg. project.version) are replaced
first by maven.
+run cp -p -R "${ROOT}/hadoop-ozone/dist/target/compose" .
Review comment:
Sure. It's a good question.
Until now we used wildcard in the docker-compose.yaml to define the
classpath, independent from the current version:
```
HADOOP_CLASSPATH:
/opt/ozone/share/ozone/lib/hadoop-ozone-filesystem-lib-current*.jar
```
Unfortunately it didn't work all the time. When I executed a command from
the container (after `docker-compose exec scm`) the jar file is not added to
the classpath.
I modified the docker-compose file to use the exact jar file. To make it
version independent instead of the simple copy I replace the current version
with maven (filtering) and copy to preprocessed version to the final
destination with dist-layout-stitching script.
Old version: src/main/compose ---[copy with dist-layout-stitching]--->
destination dir
New version: src/main/compose ---[copy and replace version with maven]--->
target/compose ---[dist layout stiching] --> destination dir
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 221339)
Time Spent: 4h (was: 3h 50m)
> OzoneFileSystem can't work with spark/hadoop2.7 because incompatible security
> classes
> -
>
> Key: HDDS-1333
> URL: https://issues.apache.org/jira/browse/HDDS-1333
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> The current ozonefs compatibility layer is broken by: HDDS-1299.
> The spark jobs (including hadoop 2.7) can't be executed any more:
> {code}
> 2019-03-25 09:50:08 INFO StateStoreCoordinatorRef:54 - Registered
> StateStoreCoordinator endpoint
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/crypto/key/KeyProviderTokenIssuer
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
> at
[jira] [Updated] (HDDS-1322) Hugo errors when building Ozone
[
https://issues.apache.org/jira/browse/HDDS-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-1322:
Resolution: Fixed
Fix Version/s: 0.4.0
Target Version/s: (was: 0.5.0)
Status: Resolved (was: Patch Available)
I've committed this via GitHub. Thank you for fixing this [~adoroszlai]!
> Hugo errors when building Ozone
> ---
>
> Key: HDDS-1322
> URL: https://issues.apache.org/jira/browse/HDDS-1322
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Arpit Agarwal
>Assignee: Doroszlai, Attila
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.0
>
> Attachments: HDDS-1322.001.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I see some odd hugo errors when building Ozone, even though I am not building
> docs.
> {code}
> $ mvn -B -q clean compile install -DskipTests=true -Dmaven.javadoc.skip=true
> -Dmaven.site.skip=true -DskipShade -Phdds
> Error: unknown command "0.4.0-SNAPSHOT" for "hugo"
> Run 'hugo --help' for usage.
> .../hadoop-hdds/docs/target
> Error: unknown command "0.4.0-SNAPSHOT" for "hugo"
> Run 'hugo --help' for usage.
> .../hadoop-hdds/docs/target
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1322) Hugo errors when building Ozone
[
https://issues.apache.org/jira/browse/HDDS-1322?focusedWorklogId=221337=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221337
]
ASF GitHub Bot logged work on HDDS-1322:
Author: ASF GitHub Bot
Created on: 01/Apr/19 15:52
Start Date: 01/Apr/19 15:52
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #671: HDDS-1322. Hugo
errors when building Ozone
URL: https://github.com/apache/hadoop/pull/671
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 221337)
Time Spent: 50m (was: 40m)
> Hugo errors when building Ozone
> ---
>
> Key: HDDS-1322
> URL: https://issues.apache.org/jira/browse/HDDS-1322
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Arpit Agarwal
>Assignee: Doroszlai, Attila
>Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-1322.001.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I see some odd hugo errors when building Ozone, even though I am not building
> docs.
> {code}
> $ mvn -B -q clean compile install -DskipTests=true -Dmaven.javadoc.skip=true
> -Dmaven.site.skip=true -DskipShade -Phdds
> Error: unknown command "0.4.0-SNAPSHOT" for "hugo"
> Run 'hugo --help' for usage.
> .../hadoop-hdds/docs/target
> Error: unknown command "0.4.0-SNAPSHOT" for "hugo"
> Run 'hugo --help' for usage.
> .../hadoop-hdds/docs/target
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1322) Hugo errors when building Ozone
[
https://issues.apache.org/jira/browse/HDDS-1322?focusedWorklogId=221336=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221336
]
ASF GitHub Bot logged work on HDDS-1322:
Author: ASF GitHub Bot
Created on: 01/Apr/19 15:51
Start Date: 01/Apr/19 15:51
Worklog Time Spent: 10m
Work Description: arp7 commented on issue #671: HDDS-1322. Hugo errors
when building Ozone
URL: https://github.com/apache/hadoop/pull/671#issuecomment-478636455
+1 LGTM
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 221336)
Time Spent: 40m (was: 0.5h)
> Hugo errors when building Ozone
> ---
>
> Key: HDDS-1322
> URL: https://issues.apache.org/jira/browse/HDDS-1322
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Arpit Agarwal
>Assignee: Doroszlai, Attila
>Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-1322.001.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I see some odd hugo errors when building Ozone, even though I am not building
> docs.
> {code}
> $ mvn -B -q clean compile install -DskipTests=true -Dmaven.javadoc.skip=true
> -Dmaven.site.skip=true -DskipShade -Phdds
> Error: unknown command "0.4.0-SNAPSHOT" for "hugo"
> Run 'hugo --help' for usage.
> .../hadoop-hdds/docs/target
> Error: unknown command "0.4.0-SNAPSHOT" for "hugo"
> Run 'hugo --help' for usage.
> .../hadoop-hdds/docs/target
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14369) RBF: Fix trailing "/" for webhdfs
[
https://issues.apache.org/jira/browse/HDFS-14369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806926#comment-16806926
]
Ayush Saxena commented on HDFS-14369:
-
Thanx [~ajisakaa] .
Guess, It isn't covering the case where the number of trailing slashes at end
is more than 1 (/user ---> Fetches in end /user/) so 1 leading slash stays.
Checked the add mount Command part how this problem isn't there, Realized as
part of add we call \{{normalizeFileSystemPath(mount)}} twice. Once in
RouterAdmin L482 and once as part of newInstance L493. So in the first time
/user/ changes to /user/ and in the second time to /user and fetches the
intended result.
The logic there fails due to \{{normalizePath(path)}} in Path.java. The javadoc
of the method says it shall remove all trailing slashes.
{code:java}
Normalize a path string to use non-duplicated forward slashes as
* the path separator and remove any trailing path separators.
{code}
But tends to remove only one here.
{code:java}
path = path.substring(0, path.length()-1);
{code}
I changed this logic for testing purpose to :
{code:java}
path = path.replaceAll("/+$", "");{code}
And the failed tests passes for me. Not sure, if we are using this utility
correct or if there is a miss in the logic here only. Ideally I think it should
remove all trailing slashes rather than just one as javadoc says.
Anyway if this a problem and I haven't misunderstood and we decide to correct
this Path problem we need to do this separately, not here in branch.
> RBF: Fix trailing "/" for webhdfs
> -
>
> Key: HDFS-14369
> URL: https://issues.apache.org/jira/browse/HDFS-14369
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-14369-HDFS-13891-regressiontest-001.patch,
> HDFS-14369-HDFS-13891.001.patch, HDFS-14369-HDFS-13891.002.patch,
> HDFS-14369-HDFS-13891.003.patch, HDFS-14369-HDFS-13891.004.patch
>
>
> WebHDFS doesn't trim trailing slash causing discrepancy in operations.
> Example below
> --
> Using HDFS API, two directory are listed.
> {code}
> $ hdfs dfs -ls hdfs://:/tmp/
> Found 2 items
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp1
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp2
> {code}
> Using WebHDFS API, only one directory is listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp/?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16387,"group":"supergroup","length":0,"modificationTime":1552016766769,"owner":"hdfs","pathSuffix":"tmp1","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
> ]}}
> {code}
> The mount table is as follows:
> {code}
> $ hdfs dfsrouteradmin -ls /tmp
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /tmp ns1->/tmp aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp1 ns1->/tmp/tmp1aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp2 ns2->/tmp/tmp2aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> {code}
> Without trailing thrash, two directories are listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":1541753421917,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753421917,"owner":"hdfs","pathSuffix":"tmp1","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"},
> {"accessTime":1541753429812,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753429812,"owner":"hdfs","pathSuffix":"tmp2","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"}
> ]}}
> {code}
> [~ajisakaa] Thanks for reporting this, I borrowed the text from
> HDFS-13972
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[
https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806924#comment-16806924
]
Erik Krogen commented on HDFS-14370:
Copying [~starphin]'s comment from HDFS-14276:
{quote}
Maybe we can just sleep 10ms in rpc server side if there are no edit logs. As a
consequence that rpc thread will be blocked for 10ms.
{quote}
I think by "server side" you're referring to the JournalNode. I think I would
prefer to keep the sleep on the NN side, to avoid handler threads being tied up
on the JN. This is similar to how the NN, when it wants a client to backoff due
to load, throws a backoff exception and expects the client to act accordingly.
> Edit log tailing fast-path should allow for backoff
> ---
>
> Key: HDFS-14370
> URL: https://issues.apache.org/jira/browse/HDFS-14370
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, qjm
>Affects Versions: 3.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
>
> As part of HDFS-13150, in-progress edit log tailing was changed to use an
> RPC-based mechanism, thus allowing the edit log tailing frequency to be
> turned way down, and allowing standby/observer NameNodes to be only a few
> milliseconds stale as compared to the Active NameNode.
> When there is a high volume of transactions on the system, each RPC fetches
> transactions and takes some time to process them, self-rate-limiting how
> frequently an RPC is submitted. In a lightly loaded cluster, however, most of
> these RPCs return an empty set of transactions, consuming a high
> (de)serialization overhead for very little benefit. This was reported by
> [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where
> the SbNN was submitting 8000 RPCs per second that returned empty.
> I propose we add some sort of backoff to the tailing, so that if an empty
> response is received, it will wait a longer period of time before submitting
> a new RPC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[ https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-14370: --- Description: As part of HDFS-13150, in-progress edit log tailing was changed to use an RPC-based mechanism, thus allowing the edit log tailing frequency to be turned way down, and allowing standby/observer NameNodes to be only a few milliseconds stale as compared to the Active NameNode. When there is a high volume of transactions on the system, each RPC fetches transactions and takes some time to process them, self-rate-limiting how frequently an RPC is submitted. In a lightly loaded cluster, however, most of these RPCs return an empty set of transactions, consuming a high (de)serialization overhead for very little benefit. This was reported by [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where the SbNN was submitting 8000 RPCs per second that returned empty. I propose we add some sort of backoff to the tailing, so that if an empty response is received, it will wait a longer period of time before submitting a new RPC. was: As part of HDFS-13150, in-progress edit log tailing was changed to use an RPC-based mechanism, thus allowing the edit log tailing frequency to be turned way down, and allowing standby/observer NameNodes to be only a few milliseconds stale as compared to the Active NameNode. When there is a high volume of transactions on the system, each RPC fetches transactions and takes some time to process them, self-rate-limiting how frequently an RPC is submitted. In a lightly loaded cluster, however, most of these RPCs return an empty set of transactions, consuming a high (de)serialization overhead for very little benefit. This was reported by [~jojochuang] in HDFS-14276 and I have also reported it on a test cluster where the SbNN was submitting 8000 RPCs per second that returned empty. I propose we add some sort of backoff to the tailing, so that if an empty response is received, it will wait a longer period of time before submitting a new RPC. > Edit log tailing fast-path should allow for backoff > --- > > Key: HDFS-14370 > URL: https://issues.apache.org/jira/browse/HDFS-14370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, qjm >Affects Versions: 3.3.0 >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > > As part of HDFS-13150, in-progress edit log tailing was changed to use an > RPC-based mechanism, thus allowing the edit log tailing frequency to be > turned way down, and allowing standby/observer NameNodes to be only a few > milliseconds stale as compared to the Active NameNode. > When there is a high volume of transactions on the system, each RPC fetches > transactions and takes some time to process them, self-rate-limiting how > frequently an RPC is submitted. In a lightly loaded cluster, however, most of > these RPCs return an empty set of transactions, consuming a high > (de)serialization overhead for very little benefit. This was reported by > [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where > the SbNN was submitting 8000 RPCs per second that returned empty. > I propose we add some sort of backoff to the tailing, so that if an empty > response is received, it will wait a longer period of time before submitting > a new RPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
