[jira] [Work logged] (HDDS-1442) add spark container to ozonesecure-mr compose files
[ https://issues.apache.org/jira/browse/HDDS-1442?focusedWorklogId=228822=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228822 ] ASF GitHub Bot logged work on HDDS-1442: Author: ASF GitHub Bot Created on: 17/Apr/19 05:25 Start Date: 17/Apr/19 05:25 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #746: HDDS-1442. add spark container to ozonesecure-mr compose files. Contributed by Ajay Kumar. URL: https://github.com/apache/hadoop/pull/746#issuecomment-483941808 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 26 | Docker mode activated. | ||| _ Prechecks _ | | 0 | yamllint | 0 | yamllint was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1037 | trunk passed | | +1 | compile | 68 | trunk passed | | +1 | mvnsite | 31 | trunk passed | | +1 | shadedclient | 675 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 18 | trunk passed | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 22 | dist in the patch failed. | | +1 | compile | 21 | the patch passed | | +1 | javac | 21 | the patch passed | | +1 | mvnsite | 22 | the patch passed | | +1 | shellcheck | 0 | There were no new shellcheck issues. | | +1 | shelldocs | 21 | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 750 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 21 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 22 | dist in the patch passed. | | +1 | asflicense | 33 | The patch does not generate ASF License warnings. | | | | 2903 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-746/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/746 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient yamllint shellcheck shelldocs | | uname | Linux 607d2209fcb4 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 04c0437 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | shellcheck | v0.4.6 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-746/1/artifact/out/patch-mvninstall-hadoop-ozone_dist.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-746/1/testReport/ | | Max. process+thread count | 446 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/dist U: hadoop-ozone/dist | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-746/1/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228822) Time Spent: 20m (was: 10m) > add spark container to ozonesecure-mr compose files > --- > > Key: HDDS-1442 > URL: https://issues.apache.org/jira/browse/HDDS-1442 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > add spark container to ozonesecure-mr compose files -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14430) RBF: Fix MockNamenode bug about mocking RPC getListing and mkdir
[ https://issues.apache.org/jira/browse/HDFS-14430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819724#comment-16819724 ] He Xiaoqiao commented on HDFS-14430: [~elgoiri],[~ayushtkn] Thanks for point out that, it makes sense for me to fix it in HDFS-14117. I will watch that issue and will close this one later. Thanks. > RBF: Fix MockNamenode bug about mocking RPC getListing and mkdir > > > Key: HDFS-14430 > URL: https://issues.apache.org/jira/browse/HDFS-14430 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: He Xiaoqiao >Assignee: He Xiaoqiao >Priority: Major > Attachments: HDFS-14430-HDFS-13891.001.patch > > > Some unexpected result when invoke mocking #getListing and #mkdirs in current > MockNamenode implement. > * for mock mkdirs, we do not check if parent directory exists. > * for mock getListing, some child dirs/files are not listing. > It may be cause some unexpected result and cause some unit test fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14433) Remove the extra empty space in the DataStreamer logging
[ https://issues.apache.org/jira/browse/HDFS-14433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yishuang Lu updated HDFS-14433: --- Affects Version/s: 3.1.2 > Remove the extra empty space in the DataStreamer logging > > > Key: HDFS-14433 > URL: https://issues.apache.org/jira/browse/HDFS-14433 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs >Affects Versions: 3.1.2 >Reporter: Yishuang Lu >Priority: Trivial > Attachments: HDFS-14433.001.patch > > > Remove the extra empty space in the DataStreamer logging -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14433) Remove the extra empty space in the DataStreamer logging
[ https://issues.apache.org/jira/browse/HDFS-14433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yishuang Lu updated HDFS-14433: --- Attachment: HDFS-14433.001.patch Status: Patch Available (was: Open) > Remove the extra empty space in the DataStreamer logging > > > Key: HDFS-14433 > URL: https://issues.apache.org/jira/browse/HDFS-14433 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs >Reporter: Yishuang Lu >Priority: Trivial > Attachments: HDFS-14433.001.patch > > > Remove the extra empty space in the DataStreamer logging -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1445) Add handling of NotReplicatedException in OzoneClient
Mukul Kumar Singh created HDDS-1445:
---
Summary: Add handling of NotReplicatedException in OzoneClient
Key: HDDS-1445
URL: https://issues.apache.org/jira/browse/HDDS-1445
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Mukul Kumar Singh
In MiniOzoneChaosCluster some of the calls fail with NotReplicatedException.
This Exception needs to be handled in OzoneClient
{code}
2019-04-17 10:13:47,254 INFO client.GrpcClientProtocolService
(GrpcClientProtocolService.java:lambda$processClientRequest$0(264)) - Failed
RaftClientRequest:client-43B95E0E3BE0->1ebec547-8cf8-4466-bf43-ea9f19fb546b@group-1B28E0BF6CBC,
cid=800, seq=0, Watch-ALL_COMMITTED(234), Message:,
reply=RaftClientReply:client-43B95E0E3BE0->1ebec547-8cf8-4466-bf43-ea9f19fb546b@group-1B28E0BF6CBC,
cid=800, FAILED org.apache.ratis.protocol.NotReplicatedException: Request with
call Id 800 and log index 234 is not yet replicated to ALL_COMMITTED,
logIndex=234, commits[1ebec547-8cf8-4466-bf43-ea9f19fb546b:c267,
7b200ef5-7711-437d-a9bc-ad0e18fdf6bb:c267,
ffbfb65f-a622-466d-b6e8-47038cc15e0b:c226]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14433) Remove the extra empty space in the DataStreamer logging
Yishuang Lu created HDFS-14433: -- Summary: Remove the extra empty space in the DataStreamer logging Key: HDFS-14433 URL: https://issues.apache.org/jira/browse/HDFS-14433 Project: Hadoop HDFS Issue Type: Task Components: hdfs Reporter: Yishuang Lu Remove the extra empty space in the DataStreamer logging -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1444) Allocate block fails in MiniOzoneChaosCluster because of InsufficientDatanodesException
Mukul Kumar Singh created HDDS-1444:
---
Summary: Allocate block fails in MiniOzoneChaosCluster because of
InsufficientDatanodesException
Key: HDDS-1444
URL: https://issues.apache.org/jira/browse/HDDS-1444
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: SCM
Affects Versions: 0.3.0
Reporter: Mukul Kumar Singh
MiniOzoneChaosCluster is failing with InsufficientDatanodesException while
writing keys to the Ozone Cluster
{code}
org.apache.hadoop.hdds.scm.pipeline.InsufficientDatanodesException: Cannot
create pipeline of factor 3 using 2 nodes.
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1443) OverlappingFileLockException exception in datanodes after restart in MiniOzoneChaosCluster
Mukul Kumar Singh created HDDS-1443:
---
Summary: OverlappingFileLockException exception in datanodes after
restart in MiniOzoneChaosCluster
Key: HDDS-1443
URL: https://issues.apache.org/jira/browse/HDDS-1443
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: Ozone Datanode
Affects Versions: 0.3.0
Reporter: Mukul Kumar Singh
Following exception can be seen in MiniOzoneChaosCluster after Datanode
restarts.
{code}
java.io.IOException: Failed to lock storage
/Users/msingh/code/apache/ozone/oz_new1/hadoop-ozone/integration-test/target/test/data/MiniOzoneClusterImpl-6e7bf33e-55b8-4437-8a0d-0da730510c70/datanode-4/data/ratis/49740cb4-b1cf-418a-af8e-1b28e0bf6cbc.
The directory is already locked
at
org.apache.ratis.server.storage.RaftStorageDirectory.tryLock(RaftStorageDirectory.java:334)
at
org.apache.ratis.server.storage.RaftStorageDirectory.lock(RaftStorageDirectory.java:291)
at
org.apache.ratis.server.storage.RaftStorageDirectory.analyzeStorage(RaftStorageDirectory.java:264)
at
org.apache.ratis.server.storage.RaftStorage.analyzeAndRecoverStorage(RaftStorage.java:91)
at
org.apache.ratis.server.storage.RaftStorage.(RaftStorage.java:59)
at org.apache.ratis.server.impl.ServerState.(ServerState.java:106)
at
org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:101)
at
org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$2(RaftServerProxy.java:208)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.OverlappingFileLockException
at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1108)
at java.nio.channels.FileChannel.tryLock(FileChannel.java:1155)
at
org.apache.ratis.server.storage.RaftStorageDirectory.tryLock(RaftStorageDirectory.java:322)
{code]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1442) add spark container to ozonesecure-mr compose files
[ https://issues.apache.org/jira/browse/HDDS-1442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-1442: - Status: Patch Available (was: Open) > add spark container to ozonesecure-mr compose files > --- > > Key: HDDS-1442 > URL: https://issues.apache.org/jira/browse/HDDS-1442 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > add spark container to ozonesecure-mr compose files -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1442) add spark container to ozonesecure-mr compose files
[ https://issues.apache.org/jira/browse/HDDS-1442?focusedWorklogId=228806=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228806 ] ASF GitHub Bot logged work on HDDS-1442: Author: ASF GitHub Bot Created on: 17/Apr/19 04:35 Start Date: 17/Apr/19 04:35 Worklog Time Spent: 10m Work Description: ajayydv commented on pull request #746: HDDS-1442. add spark container to ozonesecure-mr compose files. Contributed by Ajay Kumar. URL: https://github.com/apache/hadoop/pull/746 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228806) Time Spent: 10m Remaining Estimate: 0h > add spark container to ozonesecure-mr compose files > --- > > Key: HDDS-1442 > URL: https://issues.apache.org/jira/browse/HDDS-1442 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > add spark container to ozonesecure-mr compose files -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1442) add spark container to ozonesecure-mr compose files
[ https://issues.apache.org/jira/browse/HDDS-1442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1442: - Labels: pull-request-available (was: ) > add spark container to ozonesecure-mr compose files > --- > > Key: HDDS-1442 > URL: https://issues.apache.org/jira/browse/HDDS-1442 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: pull-request-available > > add spark container to ozonesecure-mr compose files -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1376) Datanode exits while executing client command when scmId is null
[
https://issues.apache.org/jira/browse/HDDS-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HDDS-1376:
---
Fix Version/s: 0.5.0
> Datanode exits while executing client command when scmId is null
>
>
> Key: HDDS-1376
> URL: https://issues.apache.org/jira/browse/HDDS-1376
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: MiniOzoneChaosCluster, pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Ozone Datanode exits with the following error, this happens because DN hasn't
> received a scmID from the SCM after registration but is processing a client
> command.
> {code}
> 2019-04-03 17:02:10,958 ERROR storage.RaftLogWorker
> (ExitUtils.java:terminate(133)) - Terminating with exit status 1:
> df6b578e-8d35-44f5-9b21-db7184dcc54e-RaftLogWorker failed.
> java.io.IOException: java.lang.NullPointerException: scmId cannot be null
> at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:54)
> at org.apache.ratis.util.IOUtils.toIOException(IOUtils.java:61)
> at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:83)
> at
> org.apache.ratis.server.storage.RaftLogWorker$StateMachineDataPolicy.getFromFuture(RaftLogWorker.java:76)
> at
> org.apache.ratis.server.storage.RaftLogWorker$WriteLog.execute(RaftLogWorker.java:354)
> at
> org.apache.ratis.server.storage.RaftLogWorker.run(RaftLogWorker.java:219)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException: scmId cannot be null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:110)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleCreateContainer(KeyValueHandler.java:243)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:165)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.createContainer(HddsDispatcher.java:350)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:224)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:149)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:347)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:354)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$handleWriteChunk$0(ContainerStateMachine.java:385)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run$$$capture(CompletableFuture.java:1590)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ... 1 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1442) add spark container to ozonesecure-mr compose files
Ajay Kumar created HDDS-1442: Summary: add spark container to ozonesecure-mr compose files Key: HDDS-1442 URL: https://issues.apache.org/jira/browse/HDDS-1442 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Ajay Kumar Assignee: Ajay Kumar add spark container to ozonesecure-mr compose files -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14432) dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml
Li Yong created HDFS-14432: -- Summary: dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml Key: HDFS-14432 URL: https://issues.apache.org/jira/browse/HDFS-14432 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs Reporter: Li Yong property "dfs.datanode.shared.file.descriptor.paths" appeared twice in hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml (after HDFS-6007、2.5.0) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[
https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819692#comment-16819692
]
Hadoop QA commented on HDFS-14431:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
3s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 17s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch
generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 23m
4s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14431 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966174/HDFS-14431-HDFS-13891.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a0e604e71547 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / bd3161e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26654/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26654/testReport/ |
| Max. process+thread count | 1359 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
[jira] [Commented] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[
https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819681#comment-16819681
]
Hadoop QA commented on HDFS-14431:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
10s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 18s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch
generated 4 new + 7 unchanged - 0 fixed = 11 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 39s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterRpc |
| | hadoop.hdfs.server.federation.router.TestRouterWithSecureStartup |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14431 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966167/HDFS-14431-HDFS-13891.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 99fa592af680 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / bd3161e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26653/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26653/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| Test
[jira] [Comment Edited] (HDFS-13596) NN restart fails after RollingUpgrade from 2.x to 3.x
[
https://issues.apache.org/jira/browse/HDFS-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818740#comment-16818740
]
zhouguangwei edited comment on HDFS-13596 at 4/17/19 2:44 AM:
--
after rollingUpgrade NN nodes to 3.x and keep DN 2.x , at this point, use 2.x
client to read or write data to hdfs will failure
write failure sample:
{color:#d04437}19/04/16 15:21:42 INFO hdfs.DataStreamer: Exception in
createBlockOutputStream{color}
{color:#d04437}xxx
org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException: Got
access token error, status message , ack with firstBadLink as x.x.x.x:x{color}
{color:#d04437} at
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1823){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1724){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:713){color}
{color:#d04437}x.x.x.x 19/04/16 15:21:42 WARN hdfs.DataStreamer: Abandoning
BP-1321128176-x-1552442036118:blk_1073742246_1422{color}
{color:#d04437}.x.x.x 19/04/16 15:21:42 WARN hdfs.DataStreamer: Excluding
datanode
DatanodeInfoWithStorage[x.x.x.x:x,DS-63920a14-79b9-497a-b741-21bdf1401ad1,DISK]{color}
{color:#d04437}19/04/16 15:21:42 INFO hdfs.DataStreamer: Exception in
createBlockOutputStream{color}
was (Author: zgw):
after rollingUpgrade NN nodes to 3.x and keep DN 2.x , at this point, use 2.x
client to read or write data to hdfs will failure
write failure sample:
{color:#d04437}19/04/16 15:21:42 INFO hdfs.DataStreamer: Exception in
createBlockOutputStream{color}
{color:#d04437}xxx
org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException: Got
access token error, status message , ack with firstBadLink as x.x.x.x:x{color}
{color:#d04437} at
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1823){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1724){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:713){color}
{color:#d04437}x.x.x.x 19/04/16 15:21:42 WARN hdfs.DataStreamer: Abandoning
BP-1321128176-x-1552442036118:blk_1073742246_1422{color}
{color:#d04437}.x.x.x 19/04/16 15:21:42 WARN hdfs.DataStreamer: Excluding
datanode
DatanodeInfoWithStorage[x:25009,DS-63920a14-79b9-497a-b741-21bdf1401ad1,DISK]{color}
{color:#d04437}19/04/16 15:21:42 INFO hdfs.DataStreamer: Exception in
createBlockOutputStream{color}
> NN restart fails after RollingUpgrade from 2.x to 3.x
> -
>
> Key: HDFS-13596
> URL: https://issues.apache.org/jira/browse/HDFS-13596
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Hanisha Koneru
>Assignee: Fei Hui
>Priority: Critical
> Attachments: HDFS-13596.001.patch, HDFS-13596.002.patch,
> HDFS-13596.003.patch, HDFS-13596.004.patch, HDFS-13596.005.patch,
> HDFS-13596.006.patch, HDFS-13596.007.patch
>
>
> After rollingUpgrade NN from 2.x and 3.x, if the NN is restarted, it fails
> while replaying edit logs.
> * After NN is started with rollingUpgrade, the layoutVersion written to
> editLogs (before finalizing the upgrade) is the pre-upgrade layout version
> (so as to support downgrade).
> * When writing transactions to log, NN writes as per the current layout
> version. In 3.x, erasureCoding bits are added to the editLog transactions.
> * So any edit log written after the upgrade and before finalizing the
> upgrade will have the old layout version but the new format of transactions.
> * When NN is restarted and the edit logs are replayed, the NN reads the old
> layout version from the editLog file. When parsing the transactions, it
> assumes that the transactions are also from the previous layout and hence
> skips parsing the erasureCoding bits.
> * This cascades into reading the wrong set of bits for other fields and
> leads to NN shutting down.
> Sample error output:
> {code:java}
> java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected
> length 16
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:74)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:86)
> at
> org.apache.hadoop.ipc.RetryCache$CacheEntryWithPayload.(RetryCache.java:163)
> at
>
[jira] [Assigned] (HDDS-1411) Add unit test to check if SCM correctly sends close commands for containers in closing state after a restart
[
https://issues.apache.org/jira/browse/HDDS-1411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Wagle reassigned HDDS-1411:
-
Assignee: Siddharth Wagle
> Add unit test to check if SCM correctly sends close commands for containers
> in closing state after a restart
>
>
> Key: HDDS-1411
> URL: https://issues.apache.org/jira/browse/HDDS-1411
> Project: Hadoop Distributed Data Store
> Issue Type: Test
> Components: test
>Reporter: Nanda kumar
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: newbie
>
> When the container is in CLOSING state, SCM keeps sending close command to
> the datanode until the container is either moved to QUASI_CLOSED or CLOSED
> state. The frequency in which the close command is sent by SCM depends on the
> property {{hdds.scm.replication.thread.interval}}.
> We have to add a test case to verify whether SCM is sending close commands
> for containers in the closing state even after a restart.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13596) NN restart fails after RollingUpgrade from 2.x to 3.x
[
https://issues.apache.org/jira/browse/HDFS-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818740#comment-16818740
]
zhouguangwei edited comment on HDFS-13596 at 4/17/19 2:41 AM:
--
after rollingUpgrade NN nodes to 3.x and keep DN 2.x , at this point, use 2.x
client to read or write data to hdfs will failure
write failure sample:
{color:#d04437}19/04/16 15:21:42 INFO hdfs.DataStreamer: Exception in
createBlockOutputStream{color}
{color:#d04437}xxx
org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException: Got
access token error, status message , ack with firstBadLink as x.x.x.x:x{color}
{color:#d04437} at
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1823){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1724){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:713){color}
{color:#d04437}x.x.x.x 19/04/16 15:21:42 WARN hdfs.DataStreamer: Abandoning
BP-1321128176-x-1552442036118:blk_1073742246_1422{color}
{color:#d04437}.x.x.x 19/04/16 15:21:42 WARN hdfs.DataStreamer: Excluding
datanode
DatanodeInfoWithStorage[x:25009,DS-63920a14-79b9-497a-b741-21bdf1401ad1,DISK]{color}
{color:#d04437}19/04/16 15:21:42 INFO hdfs.DataStreamer: Exception in
createBlockOutputStream{color}
was (Author: zgw):
after rollingUpgrade NN nodes to 3.x and keep DN 2.x , at this point, use 2.x
client to read or write data to hdfs will failure
write failure sample:
{color:#d04437}19/04/16 15:21:42 INFO hdfs.DataStreamer: Exception in
createBlockOutputStream{color}
{color:#d04437}org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException:
Got access token error, status message , ack with firstBadLink as
x.x.x.x:25009{color}
{color:#d04437} at
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1823){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1724){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:713){color}
{color:#d04437}x.x.x.x 19/04/16 15:21:42 WARN hdfs.DataStreamer: Abandoning
BP-1321128176-x-1552442036118:blk_1073742246_1422{color}
{color:#d04437}.x.x.x 19/04/16 15:21:42 WARN hdfs.DataStreamer: Excluding
datanode
DatanodeInfoWithStorage[x:25009,DS-63920a14-79b9-497a-b741-21bdf1401ad1,DISK]{color}
{color:#d04437}19/04/16 15:21:42 INFO hdfs.DataStreamer: Exception in
createBlockOutputStream{color}
> NN restart fails after RollingUpgrade from 2.x to 3.x
> -
>
> Key: HDFS-13596
> URL: https://issues.apache.org/jira/browse/HDFS-13596
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Hanisha Koneru
>Assignee: Fei Hui
>Priority: Critical
> Attachments: HDFS-13596.001.patch, HDFS-13596.002.patch,
> HDFS-13596.003.patch, HDFS-13596.004.patch, HDFS-13596.005.patch,
> HDFS-13596.006.patch, HDFS-13596.007.patch
>
>
> After rollingUpgrade NN from 2.x and 3.x, if the NN is restarted, it fails
> while replaying edit logs.
> * After NN is started with rollingUpgrade, the layoutVersion written to
> editLogs (before finalizing the upgrade) is the pre-upgrade layout version
> (so as to support downgrade).
> * When writing transactions to log, NN writes as per the current layout
> version. In 3.x, erasureCoding bits are added to the editLog transactions.
> * So any edit log written after the upgrade and before finalizing the
> upgrade will have the old layout version but the new format of transactions.
> * When NN is restarted and the edit logs are replayed, the NN reads the old
> layout version from the editLog file. When parsing the transactions, it
> assumes that the transactions are also from the previous layout and hence
> skips parsing the erasureCoding bits.
> * This cascades into reading the wrong set of bits for other fields and
> leads to NN shutting down.
> Sample error output:
> {code:java}
> java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected
> length 16
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:74)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:86)
> at
> org.apache.hadoop.ipc.RetryCache$CacheEntryWithPayload.(RetryCache.java:163)
> at
>
[jira] [Comment Edited] (HDFS-13596) NN restart fails after RollingUpgrade from 2.x to 3.x
[
https://issues.apache.org/jira/browse/HDFS-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818740#comment-16818740
]
zhouguangwei edited comment on HDFS-13596 at 4/17/19 2:40 AM:
--
after rollingUpgrade NN nodes to 3.x and keep DN 2.x , at this point, use 2.x
client to read or write data to hdfs will failure
write failure sample:
{color:#d04437}19/04/16 15:21:42 INFO hdfs.DataStreamer: Exception in
createBlockOutputStream{color}
{color:#d04437}org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException:
Got access token error, status message , ack with firstBadLink as
x.x.x.x:25009{color}
{color:#d04437} at
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1823){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1724){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:713){color}
{color:#d04437}x.x.x.x 19/04/16 15:21:42 WARN hdfs.DataStreamer: Abandoning
BP-1321128176-x-1552442036118:blk_1073742246_1422{color}
{color:#d04437}.x.x.x 19/04/16 15:21:42 WARN hdfs.DataStreamer: Excluding
datanode
DatanodeInfoWithStorage[x:25009,DS-63920a14-79b9-497a-b741-21bdf1401ad1,DISK]{color}
{color:#d04437}19/04/16 15:21:42 INFO hdfs.DataStreamer: Exception in
createBlockOutputStream{color}
was (Author: zgw):
after rollingUpgrade NN nodes to 3.x and keep DN 2.x , at this point, use 2.x
client to read or write data to hdfs will failure
write failure sample:
{color:#d04437}19/04/16 15:21:42 INFO hdfs.DataStreamer: Exception in
createBlockOutputStream{color}
{color:#d04437}org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException:
Got access token error, status message , ack with firstBadLink as
x.x.x.x:25009{color}
{color:#d04437} at
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1823){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1724){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:713){color}
{color:#d04437}x.x.x.x 19/04/16 15:21:42 WARN hdfs.DataStreamer: Abandoning
BP-1321128176-x-1552442036118:blk_1073742246_1422{color}
{color:#d04437}19/04/16 15:21:42 WARN hdfs.DataStreamer: Excluding datanode
DatanodeInfoWithStorage[187.4.65.81:25009,DS-63920a14-79b9-497a-b741-21bdf1401ad1,DISK]{color}
{color:#d04437}19/04/16 15:21:42 INFO hdfs.DataStreamer: Exception in
createBlockOutputStream{color}
> NN restart fails after RollingUpgrade from 2.x to 3.x
> -
>
> Key: HDFS-13596
> URL: https://issues.apache.org/jira/browse/HDFS-13596
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Hanisha Koneru
>Assignee: Fei Hui
>Priority: Critical
> Attachments: HDFS-13596.001.patch, HDFS-13596.002.patch,
> HDFS-13596.003.patch, HDFS-13596.004.patch, HDFS-13596.005.patch,
> HDFS-13596.006.patch, HDFS-13596.007.patch
>
>
> After rollingUpgrade NN from 2.x and 3.x, if the NN is restarted, it fails
> while replaying edit logs.
> * After NN is started with rollingUpgrade, the layoutVersion written to
> editLogs (before finalizing the upgrade) is the pre-upgrade layout version
> (so as to support downgrade).
> * When writing transactions to log, NN writes as per the current layout
> version. In 3.x, erasureCoding bits are added to the editLog transactions.
> * So any edit log written after the upgrade and before finalizing the
> upgrade will have the old layout version but the new format of transactions.
> * When NN is restarted and the edit logs are replayed, the NN reads the old
> layout version from the editLog file. When parsing the transactions, it
> assumes that the transactions are also from the previous layout and hence
> skips parsing the erasureCoding bits.
> * This cascades into reading the wrong set of bits for other fields and
> leads to NN shutting down.
> Sample error output:
> {code:java}
> java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected
> length 16
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:74)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:86)
> at
> org.apache.hadoop.ipc.RetryCache$CacheEntryWithPayload.(RetryCache.java:163)
> at
>
[jira] [Comment Edited] (HDFS-13596) NN restart fails after RollingUpgrade from 2.x to 3.x
[
https://issues.apache.org/jira/browse/HDFS-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818740#comment-16818740
]
zhouguangwei edited comment on HDFS-13596 at 4/17/19 2:39 AM:
--
after rollingUpgrade NN nodes to 3.x and keep DN 2.x , at this point, use 2.x
client to read or write data to hdfs will failure
write failure sample:
{color:#d04437}19/04/16 15:21:42 INFO hdfs.DataStreamer: Exception in
createBlockOutputStream{color}
{color:#d04437}org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException:
Got access token error, status message , ack with firstBadLink as
x.x.x.x:25009{color}
{color:#d04437} at
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1823){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1724){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:713){color}
{color:#d04437}x.x.x.x 19/04/16 15:21:42 WARN hdfs.DataStreamer: Abandoning
BP-1321128176-x-1552442036118:blk_1073742246_1422{color}
{color:#d04437}19/04/16 15:21:42 WARN hdfs.DataStreamer: Excluding datanode
DatanodeInfoWithStorage[187.4.65.81:25009,DS-63920a14-79b9-497a-b741-21bdf1401ad1,DISK]{color}
{color:#d04437}19/04/16 15:21:42 INFO hdfs.DataStreamer: Exception in
createBlockOutputStream{color}
was (Author: zgw):
after rollingUpgrade NN nodes to 3.x and keep DN 2.x , at this point, use 2.x
client to read or write data to hdfs will failure
write failure sample:
{color:#d04437}19/04/16 15:21:42 INFO hdfs.DataStreamer: Exception in
createBlockOutputStream{color}
{color:#d04437}org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException:
Got access token error, status message , ack with firstBadLink as
187.4.65.81:25009{color}
{color:#d04437} at
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1823){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1724){color}
{color:#d04437} at
org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:713){color}
{color:#d04437}19/04/16 15:21:42 WARN hdfs.DataStreamer: Abandoning
BP-1321128176-187.4.64.197-1552442036118:blk_1073742246_1422{color}
{color:#d04437}19/04/16 15:21:42 WARN hdfs.DataStreamer: Excluding datanode
DatanodeInfoWithStorage[187.4.65.81:25009,DS-63920a14-79b9-497a-b741-21bdf1401ad1,DISK]{color}
{color:#d04437}19/04/16 15:21:42 INFO hdfs.DataStreamer: Exception in
createBlockOutputStream{color}
> NN restart fails after RollingUpgrade from 2.x to 3.x
> -
>
> Key: HDFS-13596
> URL: https://issues.apache.org/jira/browse/HDFS-13596
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Hanisha Koneru
>Assignee: Fei Hui
>Priority: Critical
> Attachments: HDFS-13596.001.patch, HDFS-13596.002.patch,
> HDFS-13596.003.patch, HDFS-13596.004.patch, HDFS-13596.005.patch,
> HDFS-13596.006.patch, HDFS-13596.007.patch
>
>
> After rollingUpgrade NN from 2.x and 3.x, if the NN is restarted, it fails
> while replaying edit logs.
> * After NN is started with rollingUpgrade, the layoutVersion written to
> editLogs (before finalizing the upgrade) is the pre-upgrade layout version
> (so as to support downgrade).
> * When writing transactions to log, NN writes as per the current layout
> version. In 3.x, erasureCoding bits are added to the editLog transactions.
> * So any edit log written after the upgrade and before finalizing the
> upgrade will have the old layout version but the new format of transactions.
> * When NN is restarted and the edit logs are replayed, the NN reads the old
> layout version from the editLog file. When parsing the transactions, it
> assumes that the transactions are also from the previous layout and hence
> skips parsing the erasureCoding bits.
> * This cascades into reading the wrong set of bits for other fields and
> leads to NN shutting down.
> Sample error output:
> {code:java}
> java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected
> length 16
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:74)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:86)
> at
> org.apache.hadoop.ipc.RetryCache$CacheEntryWithPayload.(RetryCache.java:163)
> at
>
[jira] [Commented] (HDDS-1434) TestDatanodeStateMachine is flaky
[
https://issues.apache.org/jira/browse/HDDS-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819671#comment-16819671
]
Hudson commented on HDDS-1434:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16426 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16426/])
HDDS-1434. TestDatanodeStateMachine is flaky (#740) (bharat: rev
0a87fd1464e39d754db3d639e1d954ecdec4555f)
* (edit)
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestDatanodeStateMachine.java
* (edit)
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestCloseContainerCommandHandler.java
> TestDatanodeStateMachine is flaky
> -
>
> Key: HDDS-1434
> URL: https://issues.apache.org/jira/browse/HDDS-1434
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Doroszlai, Attila
>Priority: Major
> Labels: ozone-flaky-test, pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> TestDatanodeStateMachine is flaky.
> It has failed in the following build
>
> [https://builds.apache.org/job/PreCommit-HDDS-Build/2650/artifact/out/patch-unit-hadoop-hdds.txt]
>
> [https://builds.apache.org/job/hadoop-multibranch/job/PR-661/6/artifact/out/patch-unit-hadoop-hdds_container-service.txt]
>
> [https://builds.apache.org/job/PreCommit-HDDS-Build/2635/artifact/out/patch-unit-hadoop-hdds.txt]
> Stack trace:
> {noformat}
> java.lang.Thread.State: WAITING (on object monitor)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:389)
> at
> org.apache.hadoop.ozone.container.common.TestDatanodeStateMachine.testStartStopDatanodeStateMachine(TestDatanodeStateMachine.java:166)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
>
[jira] [Commented] (HDDS-1402) Remove unused ScmBlockLocationProtocol from ObjectStoreHandler
[ https://issues.apache.org/jira/browse/HDDS-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819673#comment-16819673 ] Hudson commented on HDDS-1402: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16427 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16427/]) HDDS-1402. Remove unused ScmBlockLocationProtocol from (bharat: rev 04c0437d13cbe8474224735cc6c41d0f6ea917f9) * (edit) hadoop-ozone/objectstore-service/src/main/java/org/apache/hadoop/hdfs/server/datanode/ObjectStoreHandler.java > Remove unused ScmBlockLocationProtocol from ObjectStoreHandler > -- > > Key: HDDS-1402 > URL: https://issues.apache.org/jira/browse/HDDS-1402 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > When I analyzed the usages of the available RPC protocols in Ozone I found > that the ScmBlockLocationProtocol is not used in ObjectStore at all. > I would propose to remove it... -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1368) Cleanup old ReplicationManager code from SCM
[
https://issues.apache.org/jira/browse/HDDS-1368?focusedWorklogId=228784=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228784
]
ASF GitHub Bot logged work on HDDS-1368:
Author: ASF GitHub Bot
Created on: 17/Apr/19 02:20
Start Date: 17/Apr/19 02:20
Worklog Time Spent: 10m
Work Description: mukul1987 commented on pull request #711: HDDS-1368.
Cleanup old ReplicationManager code from SCM.
URL: https://github.com/apache/hadoop/pull/711#discussion_r276057663
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DeadNodeHandler.java
##
@@ -18,121 +18,155 @@
package org.apache.hadoop.hdds.scm.node;
-import java.util.Set;
+import java.io.IOException;
+import java.util.Optional;
import org.apache.hadoop.hdds.protocol.DatanodeDetails;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
import org.apache.hadoop.hdds.scm.container.ContainerException;
-import org.apache.hadoop.hdds.scm.container.ContainerID;
import org.apache.hadoop.hdds.scm.container.ContainerInfo;
import org.apache.hadoop.hdds.scm.container.ContainerManager;
import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
-import org.apache.hadoop.hdds.scm.container.ContainerReplica;
-import org.apache.hadoop.hdds.scm.container.replication.ReplicationRequest;
-import org.apache.hadoop.hdds.scm.events.SCMEvents;
import org.apache.hadoop.hdds.scm.node.states.NodeNotFoundException;
+import org.apache.hadoop.hdds.scm.pipeline.PipelineManager;
+import org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException;
import org.apache.hadoop.hdds.server.events.EventHandler;
import org.apache.hadoop.hdds.server.events.EventPublisher;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import static org.apache.hadoop.hdds.scm.events.SCMEvents.CLOSE_CONTAINER;
+
/**
* Handles Dead Node event.
*/
public class DeadNodeHandler implements EventHandler {
- private final ContainerManager containerManager;
-
private final NodeManager nodeManager;
+ private final PipelineManager pipelineManager;
+ private final ContainerManager containerManager;
private static final Logger LOG =
LoggerFactory.getLogger(DeadNodeHandler.class);
- public DeadNodeHandler(NodeManager nodeManager,
- ContainerManager containerManager) {
-this.containerManager = containerManager;
+ public DeadNodeHandler(final NodeManager nodeManager,
+ final PipelineManager pipelineManager,
+ final ContainerManager containerManager) {
this.nodeManager = nodeManager;
+this.pipelineManager = pipelineManager;
+this.containerManager = containerManager;
}
@Override
- public void onMessage(DatanodeDetails datanodeDetails,
- EventPublisher publisher) {
+ public void onMessage(final DatanodeDetails datanodeDetails,
+final EventPublisher publisher) {
-// TODO: check if there are any pipeline on this node and fire close
-// pipeline event
-Set ids =
-null;
try {
- ids = nodeManager.getContainers(datanodeDetails);
-} catch (NodeNotFoundException e) {
+
+ /*
+ * We should have already destroyed all the pipelines on this datanode
+ * when it was marked as stale. Destroy pipeline should also have closed
+ * all the containers on this datanode.
+ *
+ * Ideally we should not have any pipeline or OPEN containers now.
+ *
+ * To be on a safer side, we double check here and take appropriate
+ * action.
+ */
+
+ destroyPipelines(datanodeDetails);
+ closeContainers(datanodeDetails, publisher);
Review comment:
Should closeContainer happen before destroy pipeline, so that we close
container cleanly ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 228784)
Time Spent: 1h (was: 50m)
> Cleanup old ReplicationManager code from SCM
>
>
> Key: HDDS-1368
> URL: https://issues.apache.org/jira/browse/HDDS-1368
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> HDDS-1205 brings in new ReplicationManager and HDDS-1207 plugs in the new
> code, this jira is for removing the old ReplicationManager
[jira] [Work logged] (HDDS-1368) Cleanup old ReplicationManager code from SCM
[
https://issues.apache.org/jira/browse/HDDS-1368?focusedWorklogId=228783=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228783
]
ASF GitHub Bot logged work on HDDS-1368:
Author: ASF GitHub Bot
Created on: 17/Apr/19 02:20
Start Date: 17/Apr/19 02:20
Worklog Time Spent: 10m
Work Description: mukul1987 commented on pull request #711: HDDS-1368.
Cleanup old ReplicationManager code from SCM.
URL: https://github.com/apache/hadoop/pull/711#discussion_r276057036
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/command/CommandStatusReportHandler.java
##
@@ -54,32 +55,14 @@ public void onMessage(CommandStatusReportFromDatanode
report,
cmdStatusList.forEach(cmdStatus -> {
LOGGER.trace("Emitting command status for id:{} type: {}", cmdStatus
.getCmdId(), cmdStatus.getType());
- switch (cmdStatus.getType()) {
- case replicateContainerCommand:
-publisher.fireEvent(SCMEvents.REPLICATION_STATUS, new
-ReplicationStatus(cmdStatus));
-if (cmdStatus.getStatus() == CommandStatus.Status.EXECUTED) {
- publisher.fireEvent(SCMEvents.REPLICATION_COMPLETE,
- new ReplicationManager.ReplicationCompleted(
- cmdStatus.getCmdId()));
-}
-break;
- case deleteBlocksCommand:
+ if (cmdStatus.getType() == SCMCommandProto.Type.deleteBlocksCommand) {
Review comment:
DeleteBlock handler is the only user of CommandWatcher now ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 228783)
Time Spent: 50m (was: 40m)
> Cleanup old ReplicationManager code from SCM
>
>
> Key: HDDS-1368
> URL: https://issues.apache.org/jira/browse/HDDS-1368
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> HDDS-1205 brings in new ReplicationManager and HDDS-1207 plugs in the new
> code, this jira is for removing the old ReplicationManager and related code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[ https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14431: --- Attachment: HDFS-14431-HDFS-13891.002.patch > RBF: Rename with multiple subclusters should fail if no eligible locations > -- > > Key: HDFS-14431 > URL: https://issues.apache.org/jira/browse/HDFS-14431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14431-HDFS-13891.001.patch, > HDFS-14431-HDFS-13891.002.patch > > > Currently, the rename will fail with FileNotFoundException which is not clear > to the user. > The operation should fail stating the reason is that there are no eligible > destinations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1402) Remove unused ScmBlockLocationProtocol from ObjectStoreHandler
[ https://issues.apache.org/jira/browse/HDDS-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1402: - Resolution: Fixed Fix Version/s: 0.5.0 Status: Resolved (was: Patch Available) > Remove unused ScmBlockLocationProtocol from ObjectStoreHandler > -- > > Key: HDDS-1402 > URL: https://issues.apache.org/jira/browse/HDDS-1402 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > When I analyzed the usages of the available RPC protocols in Ozone I found > that the ScmBlockLocationProtocol is not used in ObjectStore at all. > I would propose to remove it... -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1402) Remove unused ScmBlockLocationProtocol from ObjectStoreHandler
[ https://issues.apache.org/jira/browse/HDDS-1402?focusedWorklogId=228781=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228781 ] ASF GitHub Bot logged work on HDDS-1402: Author: ASF GitHub Bot Created on: 17/Apr/19 02:16 Start Date: 17/Apr/19 02:16 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #707: HDDS-1402. Remove unused ScmBlockLocationProtocol from ObjectStoreHandler URL: https://github.com/apache/hadoop/pull/707 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228781) Time Spent: 1h 10m (was: 1h) > Remove unused ScmBlockLocationProtocol from ObjectStoreHandler > -- > > Key: HDDS-1402 > URL: https://issues.apache.org/jira/browse/HDDS-1402 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > When I analyzed the usages of the available RPC protocols in Ozone I found > that the ScmBlockLocationProtocol is not used in ObjectStore at all. > I would propose to remove it... -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[ https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14431: --- Attachment: (was: HDFS-14431-HDFS-13891.002.patch) > RBF: Rename with multiple subclusters should fail if no eligible locations > -- > > Key: HDFS-14431 > URL: https://issues.apache.org/jira/browse/HDFS-14431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14431-HDFS-13891.001.patch > > > Currently, the rename will fail with FileNotFoundException which is not clear > to the user. > The operation should fail stating the reason is that there are no eligible > destinations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[ https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14431: --- Attachment: HDFS-14431-HDFS-13891.002.patch > RBF: Rename with multiple subclusters should fail if no eligible locations > -- > > Key: HDFS-14431 > URL: https://issues.apache.org/jira/browse/HDFS-14431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14431-HDFS-13891.001.patch, > HDFS-14431-HDFS-13891.002.patch > > > Currently, the rename will fail with FileNotFoundException which is not clear > to the user. > The operation should fail stating the reason is that there are no eligible > destinations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1434) TestDatanodeStateMachine is flaky
[
https://issues.apache.org/jira/browse/HDDS-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-1434:
-
Resolution: Fixed
Fix Version/s: 0.5.0
Status: Resolved (was: Patch Available)
> TestDatanodeStateMachine is flaky
> -
>
> Key: HDDS-1434
> URL: https://issues.apache.org/jira/browse/HDDS-1434
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Doroszlai, Attila
>Priority: Major
> Labels: ozone-flaky-test, pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> TestDatanodeStateMachine is flaky.
> It has failed in the following build
>
> [https://builds.apache.org/job/PreCommit-HDDS-Build/2650/artifact/out/patch-unit-hadoop-hdds.txt]
>
> [https://builds.apache.org/job/hadoop-multibranch/job/PR-661/6/artifact/out/patch-unit-hadoop-hdds_container-service.txt]
>
> [https://builds.apache.org/job/PreCommit-HDDS-Build/2635/artifact/out/patch-unit-hadoop-hdds.txt]
> Stack trace:
> {noformat}
> java.lang.Thread.State: WAITING (on object monitor)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:389)
> at
> org.apache.hadoop.ozone.container.common.TestDatanodeStateMachine.testStartStopDatanodeStateMachine(TestDatanodeStateMachine.java:166)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] TestDatanodeStateMachine.testStartStopDatanodeStateMachine:166 ?
> Timeout Timed...
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail:
[jira] [Work logged] (HDDS-1434) TestDatanodeStateMachine is flaky
[
https://issues.apache.org/jira/browse/HDDS-1434?focusedWorklogId=228780=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228780
]
ASF GitHub Bot logged work on HDDS-1434:
Author: ASF GitHub Bot
Created on: 17/Apr/19 02:13
Start Date: 17/Apr/19 02:13
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #740:
[HDDS-1434] TestDatanodeStateMachine is flaky
URL: https://github.com/apache/hadoop/pull/740
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 228780)
Time Spent: 50m (was: 40m)
> TestDatanodeStateMachine is flaky
> -
>
> Key: HDDS-1434
> URL: https://issues.apache.org/jira/browse/HDDS-1434
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Doroszlai, Attila
>Priority: Major
> Labels: ozone-flaky-test, pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> TestDatanodeStateMachine is flaky.
> It has failed in the following build
>
> [https://builds.apache.org/job/PreCommit-HDDS-Build/2650/artifact/out/patch-unit-hadoop-hdds.txt]
>
> [https://builds.apache.org/job/hadoop-multibranch/job/PR-661/6/artifact/out/patch-unit-hadoop-hdds_container-service.txt]
>
> [https://builds.apache.org/job/PreCommit-HDDS-Build/2635/artifact/out/patch-unit-hadoop-hdds.txt]
> Stack trace:
> {noformat}
> java.lang.Thread.State: WAITING (on object monitor)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:389)
> at
> org.apache.hadoop.ozone.container.common.TestDatanodeStateMachine.testStartStopDatanodeStateMachine(TestDatanodeStateMachine.java:166)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
>
[jira] [Work logged] (HDDS-1434) TestDatanodeStateMachine is flaky
[
https://issues.apache.org/jira/browse/HDDS-1434?focusedWorklogId=228779=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228779
]
ASF GitHub Bot logged work on HDDS-1434:
Author: ASF GitHub Bot
Created on: 17/Apr/19 02:12
Start Date: 17/Apr/19 02:12
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on issue #740: [HDDS-1434]
TestDatanodeStateMachine is flaky
URL: https://github.com/apache/hadoop/pull/740#issuecomment-483910163
Thank You @adoroszlai for the fix.
+1 LGTM.
I will commit this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 228779)
Time Spent: 40m (was: 0.5h)
> TestDatanodeStateMachine is flaky
> -
>
> Key: HDDS-1434
> URL: https://issues.apache.org/jira/browse/HDDS-1434
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Doroszlai, Attila
>Priority: Major
> Labels: ozone-flaky-test, pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> TestDatanodeStateMachine is flaky.
> It has failed in the following build
>
> [https://builds.apache.org/job/PreCommit-HDDS-Build/2650/artifact/out/patch-unit-hadoop-hdds.txt]
>
> [https://builds.apache.org/job/hadoop-multibranch/job/PR-661/6/artifact/out/patch-unit-hadoop-hdds_container-service.txt]
>
> [https://builds.apache.org/job/PreCommit-HDDS-Build/2635/artifact/out/patch-unit-hadoop-hdds.txt]
> Stack trace:
> {noformat}
> java.lang.Thread.State: WAITING (on object monitor)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:389)
> at
> org.apache.hadoop.ozone.container.common.TestDatanodeStateMachine.testStartStopDatanodeStateMachine(TestDatanodeStateMachine.java:166)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
>
[jira] [Commented] (HDFS-13972) RBF: Support for Delegation Token (WebHDFS)
[ https://issues.apache.org/jira/browse/HDFS-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819664#comment-16819664 ] Íñigo Goiri commented on HDFS-13972: I was thinking if there was some counter we could check to see what user executed commands. However, the RPC side doesn't have it available. Not sure how to check for this easily... How easy to make the stub for the getDatanodeReport() as a cache? No need to implement the full caching but just checking for it. In the follow-up we actually track that. Actually, isn't the RouterRpcServer already caching it? Can we rely on it? > RBF: Support for Delegation Token (WebHDFS) > --- > > Key: HDFS-13972 > URL: https://issues.apache.org/jira/browse/HDFS-13972 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: HDFS-13972-HDFS-13891.001.patch, > HDFS-13972-HDFS-13891.002.patch, HDFS-13972-HDFS-13891.003.patch, > HDFS-13972-HDFS-13891.004.patch, HDFS-13972-HDFS-13891.005.patch, > HDFS-13972-HDFS-13891.006.patch, HDFS-13972-HDFS-13891.007.patch, > HDFS-13972-HDFS-13891.008.patch, HDFS-13972-HDFS-13891.009.patch, > HDFS-13972-HDFS-13891.010.patch, HDFS-13972-HDFS-13891.011.patch, > HDFS-13972-HDFS-13891.012.patch, HDFS-13972-HDFS-13891.013.patch, > TestRouterWebHDFSContractTokens.java > > > HDFS Router should support issuing HDFS delegation tokens through WebHDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-976) Support YAML format network topology cluster definition
[ https://issues.apache.org/jira/browse/HDDS-976?focusedWorklogId=228776=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228776 ] ASF GitHub Bot logged work on HDDS-976: --- Author: ASF GitHub Bot Created on: 17/Apr/19 01:45 Start Date: 17/Apr/19 01:45 Worklog Time Spent: 10m Work Description: cjjnjust commented on issue #661: HDDS-976: Parse network topology from yaml file URL: https://github.com/apache/hadoop/pull/661#issuecomment-483905287 @xiaoyuyao , I checked failed tests, it looks like not related to PR, it seems all errors are caused by `java.lang.RuntimeException: Fail to load schema file:/testptch/hadoop/hadoop-hdds/common/target/test-classes/networkTopologyTestFiles/good.xml, error:null`. while I didn't remove it in this pr. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228776) Time Spent: 2h 40m (was: 2.5h) > Support YAML format network topology cluster definition > --- > > Key: HDDS-976 > URL: https://issues.apache.org/jira/browse/HDDS-976 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Junjie Chen >Priority: Major > Labels: pull-request-available > Attachments: NetworkTopologyDefault.yaml > > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[
https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819649#comment-16819649
]
Íñigo Goiri commented on HDFS-14117:
I created HDFS-14431 to do the fixes for renaming.
Once that's in, we can discuss what to do with the Trash.
> RBF: We can only delete the files or dirs of one subcluster in a cluster with
> multiple subclusters when trash is enabled
>
>
> Key: HDFS-14117
> URL: https://issues.apache.org/jira/browse/HDFS-14117
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: venkata ramkumar
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14117-HDFS-13891.001.patch,
> HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch,
> HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch,
> HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch,
> HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch,
> HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch,
> HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch,
> HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch,
> HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch,
> HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch,
> HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch,
> HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch
>
>
> When we delete files or dirs in hdfs, it will move the deleted files or dirs
> to trash by default.
> But in the global path we can only mount one trash dir /user. So we mount
> trash dir /user of the subcluster ns1 to the global path /user. Then we can
> delete files or dirs of ns1, but when we delete the files or dirs of another
> subcluser, such as hacluster, it will be failed.
> h1. Mount Table
> ||Global path||Target nameservice||Target path||Order||Read
> only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created||
> |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42|
> |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05|
> |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota:
> -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20|
> commands:
> {noformat}
> 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/.
> 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd
> 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/.
> 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd
> 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm
> /tmp/mapred.cmd
> 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination
> parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found.
> 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd
> 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved:
> 'hdfs://router/test/hdfs.cmd' to trash at:
> hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[ https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14431: --- Attachment: HDFS-14431-HDFS-13891.001.patch > RBF: Rename with multiple subclusters should fail if no eligible locations > -- > > Key: HDFS-14431 > URL: https://issues.apache.org/jira/browse/HDFS-14431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14431-HDFS-13891.001.patch > > > Currently, the rename will fail with FileNotFoundException which is not clear > to the user. > The operation should fail stating the reason is that there are no eligible > destinations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[ https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14431: --- Attachment: (was: HDFS-14431-HDFS-13891.000.patch) > RBF: Rename with multiple subclusters should fail if no eligible locations > -- > > Key: HDFS-14431 > URL: https://issues.apache.org/jira/browse/HDFS-14431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14431-HDFS-13891.001.patch > > > Currently, the rename will fail with FileNotFoundException which is not clear > to the user. > The operation should fail stating the reason is that there are no eligible > destinations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[ https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14431: --- Status: Patch Available (was: Open) > RBF: Rename with multiple subclusters should fail if no eligible locations > -- > > Key: HDFS-14431 > URL: https://issues.apache.org/jira/browse/HDFS-14431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14431-HDFS-13891.001.patch > > > Currently, the rename will fail with FileNotFoundException which is not clear > to the user. > The operation should fail stating the reason is that there are no eligible > destinations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819646#comment-16819646
]
Hadoop QA commented on HDFS-14390:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 28s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestMultipleNNPortQOP |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14390 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966157/HDFS-14390.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 11820fc413dc 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb
13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e543c3b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26651/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26651/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Updated] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[ https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14431: --- Attachment: HDFS-14431-HDFS-13891.000.patch > RBF: Rename with multiple subclusters should fail if no eligible locations > -- > > Key: HDFS-14431 > URL: https://issues.apache.org/jira/browse/HDFS-14431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14431-HDFS-13891.000.patch > > > Currently, the rename will fail with FileNotFoundException which is not clear > to the user. > The operation should fail stating the reason is that there are no eligible > destinations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[ https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri reassigned HDFS-14431: -- Assignee: Íñigo Goiri > RBF: Rename with multiple subclusters should fail if no eligible locations > -- > > Key: HDFS-14431 > URL: https://issues.apache.org/jira/browse/HDFS-14431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14431-HDFS-13891.000.patch > > > Currently, the rename will fail with FileNotFoundException which is not clear > to the user. > The operation should fail stating the reason is that there are no eligible > destinations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13596) NN restart fails after RollingUpgrade from 2.x to 3.x
[
https://issues.apache.org/jira/browse/HDFS-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819640#comment-16819640
]
zhouguangwei edited comment on HDFS-13596 at 4/17/19 1:18 AM:
--
[~jojochuang] Yes, I have merged the latest patch that Fei Hui provided and
rebuild HDFS project, then replace hadoop-hdfs-3.1.1.jar
was (Author: zgw):
[~jojochuang] Yes, I have merge the latest patch that Fei Hui provided and
rebuild HDFS project, then replace hadoop-hdfs-3.1.1.jar
> NN restart fails after RollingUpgrade from 2.x to 3.x
> -
>
> Key: HDFS-13596
> URL: https://issues.apache.org/jira/browse/HDFS-13596
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Hanisha Koneru
>Assignee: Fei Hui
>Priority: Critical
> Attachments: HDFS-13596.001.patch, HDFS-13596.002.patch,
> HDFS-13596.003.patch, HDFS-13596.004.patch, HDFS-13596.005.patch,
> HDFS-13596.006.patch, HDFS-13596.007.patch
>
>
> After rollingUpgrade NN from 2.x and 3.x, if the NN is restarted, it fails
> while replaying edit logs.
> * After NN is started with rollingUpgrade, the layoutVersion written to
> editLogs (before finalizing the upgrade) is the pre-upgrade layout version
> (so as to support downgrade).
> * When writing transactions to log, NN writes as per the current layout
> version. In 3.x, erasureCoding bits are added to the editLog transactions.
> * So any edit log written after the upgrade and before finalizing the
> upgrade will have the old layout version but the new format of transactions.
> * When NN is restarted and the edit logs are replayed, the NN reads the old
> layout version from the editLog file. When parsing the transactions, it
> assumes that the transactions are also from the previous layout and hence
> skips parsing the erasureCoding bits.
> * This cascades into reading the wrong set of bits for other fields and
> leads to NN shutting down.
> Sample error output:
> {code:java}
> java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected
> length 16
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:74)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:86)
> at
> org.apache.hadoop.ipc.RetryCache$CacheEntryWithPayload.(RetryCache.java:163)
> at
> org.apache.hadoop.ipc.RetryCache.addCacheEntryWithPayload(RetryCache.java:322)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheEntryWithPayload(FSNamesystem.java:960)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:397)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:249)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:888)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1086)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:937)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:910)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710)
> 2018-05-17 19:10:06,522 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: java.lang.IllegalStateException: Cannot skip to less
> than the current value (=16389), where newValue=16388
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.resetLastInodeId(FSDirectory.java:1945)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:298)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:888)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1086)
> at
>
[jira] [Commented] (HDFS-14403) Cost-Based RPC FairCallQueue
[
https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819641#comment-16819641
]
Hadoop QA commented on HDFS-14403:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 22s{color} | {color:orange} root: The patch generated 13 new + 395 unchanged
- 3 fixed = 408 total (was 398) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 47s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 40s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}184m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ipc.TestRPC |
| | hadoop.ipc.TestProtoBufRpc |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestFileCreation |
| | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14403 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966152/HDFS-14403.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 616709fc44a4 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
|
[jira] [Commented] (HDFS-13596) NN restart fails after RollingUpgrade from 2.x to 3.x
[
https://issues.apache.org/jira/browse/HDFS-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819640#comment-16819640
]
zhouguangwei commented on HDFS-13596:
-
[~jojochuang] Yes, I have merge the latest patch that Fei Hui provided and
rebuild HDFS project, then replace hadoop-hdfs-3.1.1.jar
> NN restart fails after RollingUpgrade from 2.x to 3.x
> -
>
> Key: HDFS-13596
> URL: https://issues.apache.org/jira/browse/HDFS-13596
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Hanisha Koneru
>Assignee: Fei Hui
>Priority: Critical
> Attachments: HDFS-13596.001.patch, HDFS-13596.002.patch,
> HDFS-13596.003.patch, HDFS-13596.004.patch, HDFS-13596.005.patch,
> HDFS-13596.006.patch, HDFS-13596.007.patch
>
>
> After rollingUpgrade NN from 2.x and 3.x, if the NN is restarted, it fails
> while replaying edit logs.
> * After NN is started with rollingUpgrade, the layoutVersion written to
> editLogs (before finalizing the upgrade) is the pre-upgrade layout version
> (so as to support downgrade).
> * When writing transactions to log, NN writes as per the current layout
> version. In 3.x, erasureCoding bits are added to the editLog transactions.
> * So any edit log written after the upgrade and before finalizing the
> upgrade will have the old layout version but the new format of transactions.
> * When NN is restarted and the edit logs are replayed, the NN reads the old
> layout version from the editLog file. When parsing the transactions, it
> assumes that the transactions are also from the previous layout and hence
> skips parsing the erasureCoding bits.
> * This cascades into reading the wrong set of bits for other fields and
> leads to NN shutting down.
> Sample error output:
> {code:java}
> java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected
> length 16
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:74)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:86)
> at
> org.apache.hadoop.ipc.RetryCache$CacheEntryWithPayload.(RetryCache.java:163)
> at
> org.apache.hadoop.ipc.RetryCache.addCacheEntryWithPayload(RetryCache.java:322)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheEntryWithPayload(FSNamesystem.java:960)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:397)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:249)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:888)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1086)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:937)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:910)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710)
> 2018-05-17 19:10:06,522 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: java.lang.IllegalStateException: Cannot skip to less
> than the current value (=16389), where newValue=16388
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.resetLastInodeId(FSDirectory.java:1945)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:298)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:888)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1086)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632)
> at
>
[jira] [Commented] (HDFS-13972) RBF: Support for Delegation Token (WebHDFS)
[
https://issues.apache.org/jira/browse/HDFS-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819621#comment-16819621
]
Hadoop QA commented on HDFS-13972:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
39s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
5s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 23m
45s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-13972 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966155/HDFS-13972-HDFS-13891.013.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a1ac432098f1 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb
13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / bd3161e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26652/testReport/ |
| Max. process+thread count | 998 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26652/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Support for Delegation Token (WebHDFS)
> ---
>
>
[jira] [Commented] (HDDS-1428) Remove benign warning in handleCreateContainer
[
https://issues.apache.org/jira/browse/HDDS-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819616#comment-16819616
]
Shweta commented on HDDS-1428:
--
Thanks for reporting this [~arpitagarwal] and working on this [~swagle].
Just curious, why aren't we using parameterized logging here, I see the
remaining of the class does use parameterized logging?
> Remove benign warning in handleCreateContainer
> --
>
> Key: HDDS-1428
> URL: https://issues.apache.org/jira/browse/HDDS-1428
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: newbie
> Fix For: 0.5.0
>
> Attachments: HDDS-1428.01.patch
>
>
> The following log message in KeyValueHandler#handleCreateContainer can be
> removed or moved to _debug_ level.
> {code}
> // The create container request for an already existing container can
> // arrive in case the ContainerStateMachine reapplies the transaction
> // on datanode restart. Just log a warning msg here.
> LOG.warn("Container already exists." +
> "container Id " + containerID);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819585#comment-16819585
]
Ashvin commented on HDFS-14390:
---
Thanks for reviewing [~daryn].
I posted a new patch, [^HDFS-14390.005.patch]. This patch does not include
{{FSTreeWalk}} changes as they are not needed for fixing the issue.
Regarding test verification, the current test is not creating any files. This
is because in absence of the kerberos annotation,
{{TestSecureAliasMap.testSecureConnectionToAliasMap}} will fail to create
{{BlockAliasMap.Reader reader}} and the test will fail with the following
error. Accordingly I think the unit test is reliably verifying the connection.
I am inclining towards not complicating the unit test with alias map creation
details.
{{java.io.IOException: Unable to retrieve InMemoryAliasMap for block pool id
BP-1267604097-10.84.180.32-1555451080089}}{{ }}{{at
org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.InMemoryLevelDBAliasMapClient.getAliasMap(InMemoryLevelDBAliasMapClient.java:173)}}{{at
org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.InMemoryLevelDBAliasMapClient.getReader(InMemoryLevelDBAliasMapClient.java:180)}}{{at
org.apache.hadoop.hdfs.server.aliasmap.TestSecureAliasMap.testSecureConnectionToAliasMap(TestSecureAliasMap.java:198)}}
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch,
> HDFS-14390.003.patch, HDFS-14390.004.patch, HDFS-14390.005.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13972) RBF: Support for Delegation Token (WebHDFS)
[
https://issues.apache.org/jira/browse/HDFS-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819576#comment-16819576
]
CR Hota commented on HDFS-13972:
[~elgoiri] Thanks for the comment.
In terms of testing just this feature, since its not an input/output param, the
only way I tried locally is by refactoring and adding a simple check for method
name. But to prevent erroneous usage of this by developers for other methods
may be tricky.
Is there anything specific you have in mind?
{code:java}
private void setSuperUser(String method) throws IOException {
if ("getDatanodeReport".equals(method)) {
UserGroupInformation loginUser = UserGroupInformation.getLoginUser();
RouterRpcServer.setCurrentUser(loginUser);
}
}
private DatanodeInfo chooseDatanode(final Router router,
final String path, final HttpOpParam.Op op, final long openOffset,
final String excludeDatanodes) throws IOException {
// We need to get the DNs as a privileged user
final RouterRpcServer rpcServer = getRPCServer(router);
setSuperUser("getDatanodeReport");
DatanodeInfo[] dns = null;
try {
dns = rpcServer.getDatanodeReport(DatanodeReportType.LIVE);
} catch (IOException e) {
LOG.error("Cannot get the datanodes from the RPC server", e);
} finally {
// Reset ugi to remote user for remaining operations.
RouterRpcServer.resetCurrentUser();
}
{code}
> RBF: Support for Delegation Token (WebHDFS)
> ---
>
> Key: HDFS-13972
> URL: https://issues.apache.org/jira/browse/HDFS-13972
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-13972-HDFS-13891.001.patch,
> HDFS-13972-HDFS-13891.002.patch, HDFS-13972-HDFS-13891.003.patch,
> HDFS-13972-HDFS-13891.004.patch, HDFS-13972-HDFS-13891.005.patch,
> HDFS-13972-HDFS-13891.006.patch, HDFS-13972-HDFS-13891.007.patch,
> HDFS-13972-HDFS-13891.008.patch, HDFS-13972-HDFS-13891.009.patch,
> HDFS-13972-HDFS-13891.010.patch, HDFS-13972-HDFS-13891.011.patch,
> HDFS-13972-HDFS-13891.012.patch, HDFS-13972-HDFS-13891.013.patch,
> TestRouterWebHDFSContractTokens.java
>
>
> HDFS Router should support issuing HDFS delegation tokens through WebHDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashvin updated HDFS-14390:
--
Attachment: HDFS-14390.005.patch
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch,
> HDFS-14390.003.patch, HDFS-14390.004.patch, HDFS-14390.005.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13972) RBF: Support for Delegation Token (WebHDFS)
[ https://issues.apache.org/jira/browse/HDFS-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CR Hota updated HDFS-13972: --- Attachment: HDFS-13972-HDFS-13891.013.patch > RBF: Support for Delegation Token (WebHDFS) > --- > > Key: HDFS-13972 > URL: https://issues.apache.org/jira/browse/HDFS-13972 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: HDFS-13972-HDFS-13891.001.patch, > HDFS-13972-HDFS-13891.002.patch, HDFS-13972-HDFS-13891.003.patch, > HDFS-13972-HDFS-13891.004.patch, HDFS-13972-HDFS-13891.005.patch, > HDFS-13972-HDFS-13891.006.patch, HDFS-13972-HDFS-13891.007.patch, > HDFS-13972-HDFS-13891.008.patch, HDFS-13972-HDFS-13891.009.patch, > HDFS-13972-HDFS-13891.010.patch, HDFS-13972-HDFS-13891.011.patch, > HDFS-13972-HDFS-13891.012.patch, HDFS-13972-HDFS-13891.013.patch, > TestRouterWebHDFSContractTokens.java > > > HDFS Router should support issuing HDFS delegation tokens through WebHDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14403) Cost-Based RPC FairCallQueue
[
https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819542#comment-16819542
]
Christopher Gregorian commented on HDFS-14403:
--
I rebased [~daryn]'s patch on trunk in version 004, haven't made any other
changes yet. Going to work on integrating with {{CostProvider}} next :)
> Cost-Based RPC FairCallQueue
>
>
> Key: HDFS-14403
> URL: https://issues.apache.org/jira/browse/HDFS-14403
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ipc, namenode
>Reporter: Erik Krogen
>Assignee: Christopher Gregorian
>Priority: Major
> Labels: qos, rpc
> Attachments: CostBasedFairCallQueueDesign_v0.pdf,
> HDFS-14403.001.patch, HDFS-14403.002.patch, HDFS-14403.003.patch,
> HDFS-14403.004.patch, HDFS-14403.branch-2.8.patch
>
>
> HADOOP-15016 initially described extensions to the Hadoop FairCallQueue
> encompassing both cost-based analysis of incoming RPCs, as well as support
> for reservations of RPC capacity for system/platform users. This JIRA intends
> to track the former, as HADOOP-15016 was repurposed to more specifically
> focus on the reservation portion of the work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14403) Cost-Based RPC FairCallQueue
[ https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Gregorian updated HDFS-14403: - Attachment: HDFS-14403.004.patch > Cost-Based RPC FairCallQueue > > > Key: HDFS-14403 > URL: https://issues.apache.org/jira/browse/HDFS-14403 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ipc, namenode >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Labels: qos, rpc > Attachments: CostBasedFairCallQueueDesign_v0.pdf, > HDFS-14403.001.patch, HDFS-14403.002.patch, HDFS-14403.003.patch, > HDFS-14403.004.patch, HDFS-14403.branch-2.8.patch > > > HADOOP-15016 initially described extensions to the Hadoop FairCallQueue > encompassing both cost-based analysis of incoming RPCs, as well as support > for reservations of RPC capacity for system/platform users. This JIRA intends > to track the former, as HADOOP-15016 was repurposed to more specifically > focus on the reservation portion of the work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1430) NPE if secure ozone if KMS uri is not defined.
[ https://issues.apache.org/jira/browse/HDDS-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-1430: - Priority: Major (was: Blocker) > NPE if secure ozone if KMS uri is not defined. > -- > > Key: HDDS-1430 > URL: https://issues.apache.org/jira/browse/HDDS-1430 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Affects Versions: 0.4.0 >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > > OzoneKMSUtil.getKeyProvider throws NPE if KMS uri is not defined. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
Íñigo Goiri created HDFS-14431: -- Summary: RBF: Rename with multiple subclusters should fail if no eligible locations Key: HDFS-14431 URL: https://issues.apache.org/jira/browse/HDFS-14431 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Íñigo Goiri Currently, the rename will fail with FileNotFoundException which is not clear to the user. The operation should fail stating the reason is that there are no eligible destinations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-789) Reduce usage of Guava Preconditions
[ https://issues.apache.org/jira/browse/HDDS-789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819494#comment-16819494 ] Shweta commented on HDDS-789: - [~danilocop] are you working on this? > Reduce usage of Guava Preconditions > > > Key: HDDS-789 > URL: https://issues.apache.org/jira/browse/HDDS-789 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: Danilo Perez >Priority: Major > Labels: newbie > > In the spirit of minimizing Guava dependencies, we can replace most usage of > Preconditions with the > [Objects|https://docs.oracle.com/javase/8/docs/api/java/util/Objects.html] > class. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-976) Support YAML format network topology cluster definition
[ https://issues.apache.org/jira/browse/HDDS-976?focusedWorklogId=228699=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228699 ] ASF GitHub Bot logged work on HDDS-976: --- Author: ASF GitHub Bot Created on: 16/Apr/19 21:08 Start Date: 16/Apr/19 21:08 Worklog Time Spent: 10m Work Description: xiaoyuyao commented on issue #661: HDDS-976: Parse network topology from yaml file URL: https://github.com/apache/hadoop/pull/661#issuecomment-483844270 Thanks @cjjnjust for the update. Some of the unit test failures seem related, can you check/fix them? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228699) Time Spent: 2.5h (was: 2h 20m) > Support YAML format network topology cluster definition > --- > > Key: HDDS-976 > URL: https://issues.apache.org/jira/browse/HDDS-976 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Junjie Chen >Priority: Major > Labels: pull-request-available > Attachments: NetworkTopologyDefault.yaml > > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1374) ContainerStateMap cannot find container while allocating blocks.
[
https://issues.apache.org/jira/browse/HDDS-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819489#comment-16819489
]
Hudson commented on HDDS-1374:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16423 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16423/])
HDDS-1374. ContainerStateMap cannot find container while allocating (github:
rev 3ac3e50b207e27c69e26669c47b1642827c76db0)
* (edit)
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/SCMContainerManager.java
> ContainerStateMap cannot find container while allocating blocks.
>
>
> Key: HDDS-1374
> URL: https://issues.apache.org/jira/browse/HDDS-1374
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.5.0
>Reporter: Mukul Kumar Singh
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: MiniOzoneChaosCluster, pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> ContainerStateMap cannot find container while allocating blocks.
> {code}
> org.apache.hadoop.hdds.scm.container.ContainerNotFoundException: #14
> at
> org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.checkIfContainerExist(ContainerStateMap.java:542)
> at
> org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.getContainerInfo(ContainerStateMap.java:189)
> at
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.getContainer(ContainerStateManager.java:483)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getContainer(SCMContainerManager.java:195)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getContainersForOwner(SCMContainerManager.java:466)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getMatchingContainer(SCMContainerManager.java:387)
> at
> org.apache.hadoop.hdds.scm.block.BlockManagerImpl.allocateBlock(BlockManagerImpl.java:201)
> at
> org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer.allocateBlock(SCMBlockProtocolServer.java:172)
> at
> org.apache.hadoop.ozone.protocolPB.ScmBlockLocationProtocolServerSideTranslatorPB.allocateScmBlock(ScmBlockLocationProtocolServerSideTranslatorPB.java:82)
> at
> org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:7533)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1376) Datanode exits while executing client command when scmId is null
[
https://issues.apache.org/jira/browse/HDDS-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819490#comment-16819490
]
Hudson commented on HDDS-1376:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16424 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16424/])
HDDS-1376. Datanode exits while executing client command when scmId is (github:
rev e543c3b31aa607e62b7bd718d9cb7f17e6b03072)
* (edit)
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestSecureOzoneContainer.java
* (edit)
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
* (edit)
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainerWithTLS.java
* (edit)
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainer.java
* (edit)
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/VersionEndpointTask.java
* (edit)
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestCloseContainerCommandHandler.java
* (edit)
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestDatanodeStateMachine.java
* (edit)
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeStateMachine.java
> Datanode exits while executing client command when scmId is null
>
>
> Key: HDDS-1376
> URL: https://issues.apache.org/jira/browse/HDDS-1376
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: MiniOzoneChaosCluster, pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Ozone Datanode exits with the following error, this happens because DN hasn't
> received a scmID from the SCM after registration but is processing a client
> command.
> {code}
> 2019-04-03 17:02:10,958 ERROR storage.RaftLogWorker
> (ExitUtils.java:terminate(133)) - Terminating with exit status 1:
> df6b578e-8d35-44f5-9b21-db7184dcc54e-RaftLogWorker failed.
> java.io.IOException: java.lang.NullPointerException: scmId cannot be null
> at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:54)
> at org.apache.ratis.util.IOUtils.toIOException(IOUtils.java:61)
> at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:83)
> at
> org.apache.ratis.server.storage.RaftLogWorker$StateMachineDataPolicy.getFromFuture(RaftLogWorker.java:76)
> at
> org.apache.ratis.server.storage.RaftLogWorker$WriteLog.execute(RaftLogWorker.java:354)
> at
> org.apache.ratis.server.storage.RaftLogWorker.run(RaftLogWorker.java:219)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException: scmId cannot be null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:110)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleCreateContainer(KeyValueHandler.java:243)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:165)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.createContainer(HddsDispatcher.java:350)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:224)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:149)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:347)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:354)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$handleWriteChunk$0(ContainerStateMachine.java:385)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run$$$capture(CompletableFuture.java:1590)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ... 1 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HDDS-1376) Datanode exits while executing client command when scmId is null
[
https://issues.apache.org/jira/browse/HDDS-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanisha Koneru updated HDDS-1376:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Datanode exits while executing client command when scmId is null
>
>
> Key: HDDS-1376
> URL: https://issues.apache.org/jira/browse/HDDS-1376
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: MiniOzoneChaosCluster, pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Ozone Datanode exits with the following error, this happens because DN hasn't
> received a scmID from the SCM after registration but is processing a client
> command.
> {code}
> 2019-04-03 17:02:10,958 ERROR storage.RaftLogWorker
> (ExitUtils.java:terminate(133)) - Terminating with exit status 1:
> df6b578e-8d35-44f5-9b21-db7184dcc54e-RaftLogWorker failed.
> java.io.IOException: java.lang.NullPointerException: scmId cannot be null
> at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:54)
> at org.apache.ratis.util.IOUtils.toIOException(IOUtils.java:61)
> at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:83)
> at
> org.apache.ratis.server.storage.RaftLogWorker$StateMachineDataPolicy.getFromFuture(RaftLogWorker.java:76)
> at
> org.apache.ratis.server.storage.RaftLogWorker$WriteLog.execute(RaftLogWorker.java:354)
> at
> org.apache.ratis.server.storage.RaftLogWorker.run(RaftLogWorker.java:219)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException: scmId cannot be null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:110)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleCreateContainer(KeyValueHandler.java:243)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:165)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.createContainer(HddsDispatcher.java:350)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:224)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:149)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:347)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:354)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$handleWriteChunk$0(ContainerStateMachine.java:385)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run$$$capture(CompletableFuture.java:1590)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ... 1 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1376) Datanode exits while executing client command when scmId is null
[
https://issues.apache.org/jira/browse/HDDS-1376?focusedWorklogId=228682=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228682
]
ASF GitHub Bot logged work on HDDS-1376:
Author: ASF GitHub Bot
Created on: 16/Apr/19 20:51
Start Date: 16/Apr/19 20:51
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #724:
HDDS-1376. Datanode exits while executing client command when scmId is null
URL: https://github.com/apache/hadoop/pull/724
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 228682)
Time Spent: 1.5h (was: 1h 20m)
> Datanode exits while executing client command when scmId is null
>
>
> Key: HDDS-1376
> URL: https://issues.apache.org/jira/browse/HDDS-1376
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: MiniOzoneChaosCluster, pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Ozone Datanode exits with the following error, this happens because DN hasn't
> received a scmID from the SCM after registration but is processing a client
> command.
> {code}
> 2019-04-03 17:02:10,958 ERROR storage.RaftLogWorker
> (ExitUtils.java:terminate(133)) - Terminating with exit status 1:
> df6b578e-8d35-44f5-9b21-db7184dcc54e-RaftLogWorker failed.
> java.io.IOException: java.lang.NullPointerException: scmId cannot be null
> at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:54)
> at org.apache.ratis.util.IOUtils.toIOException(IOUtils.java:61)
> at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:83)
> at
> org.apache.ratis.server.storage.RaftLogWorker$StateMachineDataPolicy.getFromFuture(RaftLogWorker.java:76)
> at
> org.apache.ratis.server.storage.RaftLogWorker$WriteLog.execute(RaftLogWorker.java:354)
> at
> org.apache.ratis.server.storage.RaftLogWorker.run(RaftLogWorker.java:219)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException: scmId cannot be null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:110)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleCreateContainer(KeyValueHandler.java:243)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:165)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.createContainer(HddsDispatcher.java:350)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:224)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:149)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:347)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:354)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$handleWriteChunk$0(ContainerStateMachine.java:385)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run$$$capture(CompletableFuture.java:1590)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ... 1 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1376) Datanode exits while executing client command when scmId is null
[
https://issues.apache.org/jira/browse/HDDS-1376?focusedWorklogId=228681=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228681
]
ASF GitHub Bot logged work on HDDS-1376:
Author: ASF GitHub Bot
Created on: 16/Apr/19 20:51
Start Date: 16/Apr/19 20:51
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on issue #724: HDDS-1376.
Datanode exits while executing client command when scmId is null
URL: https://github.com/apache/hadoop/pull/724#issuecomment-483838370
Thank you @bharatviswa504 and @arp7 for the reviews. Merging with trunk.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 228681)
Time Spent: 1h 20m (was: 1h 10m)
> Datanode exits while executing client command when scmId is null
>
>
> Key: HDDS-1376
> URL: https://issues.apache.org/jira/browse/HDDS-1376
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: MiniOzoneChaosCluster, pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Ozone Datanode exits with the following error, this happens because DN hasn't
> received a scmID from the SCM after registration but is processing a client
> command.
> {code}
> 2019-04-03 17:02:10,958 ERROR storage.RaftLogWorker
> (ExitUtils.java:terminate(133)) - Terminating with exit status 1:
> df6b578e-8d35-44f5-9b21-db7184dcc54e-RaftLogWorker failed.
> java.io.IOException: java.lang.NullPointerException: scmId cannot be null
> at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:54)
> at org.apache.ratis.util.IOUtils.toIOException(IOUtils.java:61)
> at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:83)
> at
> org.apache.ratis.server.storage.RaftLogWorker$StateMachineDataPolicy.getFromFuture(RaftLogWorker.java:76)
> at
> org.apache.ratis.server.storage.RaftLogWorker$WriteLog.execute(RaftLogWorker.java:354)
> at
> org.apache.ratis.server.storage.RaftLogWorker.run(RaftLogWorker.java:219)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException: scmId cannot be null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:110)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleCreateContainer(KeyValueHandler.java:243)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:165)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.createContainer(HddsDispatcher.java:350)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:224)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:149)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:347)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:354)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$handleWriteChunk$0(ContainerStateMachine.java:385)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run$$$capture(CompletableFuture.java:1590)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ... 1 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1374) ContainerStateMap cannot find container while allocating blocks.
[
https://issues.apache.org/jira/browse/HDDS-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819477#comment-16819477
]
Bharat Viswanadham commented on HDDS-1374:
--
Thank You [~arpitagarwal] for the review.
I have committed this to the trunk.
> ContainerStateMap cannot find container while allocating blocks.
>
>
> Key: HDDS-1374
> URL: https://issues.apache.org/jira/browse/HDDS-1374
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.5.0
>Reporter: Mukul Kumar Singh
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: MiniOzoneChaosCluster, pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> ContainerStateMap cannot find container while allocating blocks.
> {code}
> org.apache.hadoop.hdds.scm.container.ContainerNotFoundException: #14
> at
> org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.checkIfContainerExist(ContainerStateMap.java:542)
> at
> org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.getContainerInfo(ContainerStateMap.java:189)
> at
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.getContainer(ContainerStateManager.java:483)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getContainer(SCMContainerManager.java:195)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getContainersForOwner(SCMContainerManager.java:466)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getMatchingContainer(SCMContainerManager.java:387)
> at
> org.apache.hadoop.hdds.scm.block.BlockManagerImpl.allocateBlock(BlockManagerImpl.java:201)
> at
> org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer.allocateBlock(SCMBlockProtocolServer.java:172)
> at
> org.apache.hadoop.ozone.protocolPB.ScmBlockLocationProtocolServerSideTranslatorPB.allocateScmBlock(ScmBlockLocationProtocolServerSideTranslatorPB.java:82)
> at
> org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:7533)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1374) ContainerStateMap cannot find container while allocating blocks.
[
https://issues.apache.org/jira/browse/HDDS-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-1374:
-
Resolution: Fixed
Fix Version/s: 0.5.0
Status: Resolved (was: Patch Available)
> ContainerStateMap cannot find container while allocating blocks.
>
>
> Key: HDDS-1374
> URL: https://issues.apache.org/jira/browse/HDDS-1374
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.5.0
>Reporter: Mukul Kumar Singh
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: MiniOzoneChaosCluster, pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> ContainerStateMap cannot find container while allocating blocks.
> {code}
> org.apache.hadoop.hdds.scm.container.ContainerNotFoundException: #14
> at
> org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.checkIfContainerExist(ContainerStateMap.java:542)
> at
> org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.getContainerInfo(ContainerStateMap.java:189)
> at
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.getContainer(ContainerStateManager.java:483)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getContainer(SCMContainerManager.java:195)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getContainersForOwner(SCMContainerManager.java:466)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getMatchingContainer(SCMContainerManager.java:387)
> at
> org.apache.hadoop.hdds.scm.block.BlockManagerImpl.allocateBlock(BlockManagerImpl.java:201)
> at
> org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer.allocateBlock(SCMBlockProtocolServer.java:172)
> at
> org.apache.hadoop.ozone.protocolPB.ScmBlockLocationProtocolServerSideTranslatorPB.allocateScmBlock(ScmBlockLocationProtocolServerSideTranslatorPB.java:82)
> at
> org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:7533)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1374) ContainerStateMap cannot find container while allocating blocks.
[
https://issues.apache.org/jira/browse/HDDS-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-1374:
-
Labels: MiniOzoneChaosCluster pull-request-available (was:
MiniOzoneChaosCluster)
> ContainerStateMap cannot find container while allocating blocks.
>
>
> Key: HDDS-1374
> URL: https://issues.apache.org/jira/browse/HDDS-1374
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.5.0
>Reporter: Mukul Kumar Singh
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: MiniOzoneChaosCluster, pull-request-available
>
> ContainerStateMap cannot find container while allocating blocks.
> {code}
> org.apache.hadoop.hdds.scm.container.ContainerNotFoundException: #14
> at
> org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.checkIfContainerExist(ContainerStateMap.java:542)
> at
> org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.getContainerInfo(ContainerStateMap.java:189)
> at
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.getContainer(ContainerStateManager.java:483)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getContainer(SCMContainerManager.java:195)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getContainersForOwner(SCMContainerManager.java:466)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getMatchingContainer(SCMContainerManager.java:387)
> at
> org.apache.hadoop.hdds.scm.block.BlockManagerImpl.allocateBlock(BlockManagerImpl.java:201)
> at
> org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer.allocateBlock(SCMBlockProtocolServer.java:172)
> at
> org.apache.hadoop.ozone.protocolPB.ScmBlockLocationProtocolServerSideTranslatorPB.allocateScmBlock(ScmBlockLocationProtocolServerSideTranslatorPB.java:82)
> at
> org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:7533)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1374) ContainerStateMap cannot find container while allocating blocks.
[
https://issues.apache.org/jira/browse/HDDS-1374?focusedWorklogId=228676=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228676
]
ASF GitHub Bot logged work on HDDS-1374:
Author: ASF GitHub Bot
Created on: 16/Apr/19 20:49
Start Date: 16/Apr/19 20:49
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on issue #735: HDDS-1374.
ContainerStateMap cannot find container while allocating blocks.
URL: https://github.com/apache/hadoop/pull/735#issuecomment-483837606
Test failures are not related to this patch.
I will commit this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 228676)
Time Spent: 10m
Remaining Estimate: 0h
> ContainerStateMap cannot find container while allocating blocks.
>
>
> Key: HDDS-1374
> URL: https://issues.apache.org/jira/browse/HDDS-1374
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.5.0
>Reporter: Mukul Kumar Singh
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: MiniOzoneChaosCluster, pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> ContainerStateMap cannot find container while allocating blocks.
> {code}
> org.apache.hadoop.hdds.scm.container.ContainerNotFoundException: #14
> at
> org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.checkIfContainerExist(ContainerStateMap.java:542)
> at
> org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.getContainerInfo(ContainerStateMap.java:189)
> at
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.getContainer(ContainerStateManager.java:483)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getContainer(SCMContainerManager.java:195)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getContainersForOwner(SCMContainerManager.java:466)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getMatchingContainer(SCMContainerManager.java:387)
> at
> org.apache.hadoop.hdds.scm.block.BlockManagerImpl.allocateBlock(BlockManagerImpl.java:201)
> at
> org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer.allocateBlock(SCMBlockProtocolServer.java:172)
> at
> org.apache.hadoop.ozone.protocolPB.ScmBlockLocationProtocolServerSideTranslatorPB.allocateScmBlock(ScmBlockLocationProtocolServerSideTranslatorPB.java:82)
> at
> org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:7533)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1374) ContainerStateMap cannot find container while allocating blocks.
[
https://issues.apache.org/jira/browse/HDDS-1374?focusedWorklogId=228677=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228677
]
ASF GitHub Bot logged work on HDDS-1374:
Author: ASF GitHub Bot
Created on: 16/Apr/19 20:49
Start Date: 16/Apr/19 20:49
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #735:
HDDS-1374. ContainerStateMap cannot find container while allocating blocks.
URL: https://github.com/apache/hadoop/pull/735
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 228677)
Time Spent: 20m (was: 10m)
> ContainerStateMap cannot find container while allocating blocks.
>
>
> Key: HDDS-1374
> URL: https://issues.apache.org/jira/browse/HDDS-1374
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.5.0
>Reporter: Mukul Kumar Singh
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: MiniOzoneChaosCluster, pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> ContainerStateMap cannot find container while allocating blocks.
> {code}
> org.apache.hadoop.hdds.scm.container.ContainerNotFoundException: #14
> at
> org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.checkIfContainerExist(ContainerStateMap.java:542)
> at
> org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.getContainerInfo(ContainerStateMap.java:189)
> at
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.getContainer(ContainerStateManager.java:483)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getContainer(SCMContainerManager.java:195)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getContainersForOwner(SCMContainerManager.java:466)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getMatchingContainer(SCMContainerManager.java:387)
> at
> org.apache.hadoop.hdds.scm.block.BlockManagerImpl.allocateBlock(BlockManagerImpl.java:201)
> at
> org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer.allocateBlock(SCMBlockProtocolServer.java:172)
> at
> org.apache.hadoop.ozone.protocolPB.ScmBlockLocationProtocolServerSideTranslatorPB.allocateScmBlock(ScmBlockLocationProtocolServerSideTranslatorPB.java:82)
> at
> org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:7533)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-746) Clearly document the various metadata and data directory locations
[ https://issues.apache.org/jira/browse/HDDS-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reassigned HDDS-746: -- Assignee: Shweta (was: Arpit Agarwal) > Clearly document the various metadata and data directory locations > -- > > Key: HDDS-746 > URL: https://issues.apache.org/jira/browse/HDDS-746 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: Shweta >Priority: Major > Labels: newbie > > We should document all the data and metadata directory locations in one place. > Currently it's not very clear how to configure them all and the fallback > locations for each when configs are missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1433) Rename GetScmInfoRespsonseProto to GetScmInfoResponseProto due to typos
[
https://issues.apache.org/jira/browse/HDDS-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819449#comment-16819449
]
Xiaoyu Yao commented on HDDS-1433:
--
Thanks [~jiwq] for fixing this. Patch LGTM, +1.
> Rename GetScmInfoRespsonseProto to GetScmInfoResponseProto due to typos
> ---
>
> Key: HDDS-1433
> URL: https://issues.apache.org/jira/browse/HDDS-1433
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: bianqi
>Assignee: Wanqiang Ji
>Priority: Major
> Labels: newbie
> Attachments: HDDS-1433.001.patch
>
>
> We got a typo in hdds.proto file
> - {{GetScmInfoRespsonseProto}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1432) Ozone client list command truncates response without any indication
[
https://issues.apache.org/jira/browse/HDDS-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819440#comment-16819440
]
Hudson commented on HDDS-1432:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16422 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16422/])
HDDS-1432. Ozone client list command truncates response without any (arp: rev
f2ab2795db0da1c912f86855031604de389411da)
* (edit)
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/keys/ListKeyHandler.java
* (edit)
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestOzoneShell.java
> Ozone client list command truncates response without any indication
> ---
>
> Key: HDDS-1432
> URL: https://issues.apache.org/jira/browse/HDDS-1432
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Affects Versions: 0.5.0
>Reporter: Siddharth Wagle
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Attachments: HDDS-1432.01.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Trying to list all keys in a bucket gives truncated results unless the limit
> override is provided as such:
> {code}
> bin/ozone --config /etc/ozone/conf sh key list -l 5000 /vol-test-1/bucket1/
> {code}
> The proposal is to provide a warning about possible truncation of results and
> the override option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1432) Ozone client list command truncates response without any indication
[
https://issues.apache.org/jira/browse/HDDS-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-1432:
Resolution: Fixed
Status: Resolved (was: Patch Available)
I've committed this. Thanks for the contribution [~swagle].
> Ozone client list command truncates response without any indication
> ---
>
> Key: HDDS-1432
> URL: https://issues.apache.org/jira/browse/HDDS-1432
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Affects Versions: 0.5.0
>Reporter: Siddharth Wagle
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Attachments: HDDS-1432.01.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Trying to list all keys in a bucket gives truncated results unless the limit
> override is provided as such:
> {code}
> bin/ozone --config /etc/ozone/conf sh key list -l 5000 /vol-test-1/bucket1/
> {code}
> The proposal is to provide a warning about possible truncation of results and
> the override option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1441) Remove usage of getRetryFailureException
[ https://issues.apache.org/jira/browse/HDDS-1441?focusedWorklogId=228616=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228616 ] ASF GitHub Bot logged work on HDDS-1441: Author: ASF GitHub Bot Created on: 16/Apr/19 19:26 Start Date: 16/Apr/19 19:26 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #745: HDDS-1441. Remove usage of getRetryFailureException. (swagle) URL: https://github.com/apache/hadoop/pull/745#issuecomment-483810540 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 25 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 820 | Maven dependency ordering for branch | | +1 | mvninstall | 1059 | trunk passed | | +1 | compile | 967 | trunk passed | | +1 | checkstyle | 140 | trunk passed | | +1 | mvnsite | 107 | trunk passed | | +1 | shadedclient | 997 | branch has no errors when building and testing our client artifacts. | | +1 | findbugs | 102 | trunk passed | | +1 | javadoc | 74 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 23 | Maven dependency ordering for patch | | +1 | mvninstall | 60 | the patch passed | | +1 | compile | 923 | the patch passed | | +1 | javac | 923 | the patch passed | | +1 | checkstyle | 142 | the patch passed | | +1 | mvnsite | 80 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 690 | patch has no errors when building and testing our client artifacts. | | +1 | findbugs | 117 | the patch passed | | +1 | javadoc | 70 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 37 | client in the patch passed. | | +1 | unit | 56 | ozone-manager in the patch passed. | | +1 | asflicense | 49 | The patch does not generate ASF License warnings. | | | | 6573 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-745/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/745 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 3feb6c7539b9 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / be6c801 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-745/2/testReport/ | | Max. process+thread count | 436 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/client hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-745/2/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228616) Time Spent: 1h (was: 50m) > Remove usage of getRetryFailureException > > > Key: HDDS-1441 > URL: https://issues.apache.org/jira/browse/HDDS-1441 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: Siddharth Wagle >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Per [~szetszwo]'s comment on RATIS-518, we can remove the usage of > getRetryFailureException. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1441) Remove usage of getRetryFailureException
[ https://issues.apache.org/jira/browse/HDDS-1441?focusedWorklogId=228615=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228615 ] ASF GitHub Bot logged work on HDDS-1441: Author: ASF GitHub Bot Created on: 16/Apr/19 19:22 Start Date: 16/Apr/19 19:22 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #745: HDDS-1441. Remove usage of getRetryFailureException. (swagle) URL: https://github.com/apache/hadoop/pull/745#issuecomment-483809489 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 32 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 276 | Maven dependency ordering for branch | | +1 | mvninstall | 1046 | trunk passed | | +1 | compile | 962 | trunk passed | | +1 | checkstyle | 130 | trunk passed | | +1 | mvnsite | 109 | trunk passed | | +1 | shadedclient | 970 | branch has no errors when building and testing our client artifacts. | | +1 | findbugs | 99 | trunk passed | | +1 | javadoc | 69 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 21 | Maven dependency ordering for patch | | +1 | mvninstall | 60 | the patch passed | | +1 | compile | 898 | the patch passed | | +1 | javac | 898 | the patch passed | | +1 | checkstyle | 136 | the patch passed | | +1 | mvnsite | 80 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 675 | patch has no errors when building and testing our client artifacts. | | +1 | findbugs | 101 | the patch passed | | +1 | javadoc | 47 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 24 | client in the patch passed. | | +1 | unit | 41 | ozone-manager in the patch passed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 5834 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-745/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/745 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 427845516e5b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / be6c801 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-745/3/testReport/ | | Max. process+thread count | 446 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/client hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-745/3/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228615) Time Spent: 50m (was: 40m) > Remove usage of getRetryFailureException > > > Key: HDDS-1441 > URL: https://issues.apache.org/jira/browse/HDDS-1441 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: Siddharth Wagle >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Per [~szetszwo]'s comment on RATIS-518, we can remove the usage of > getRetryFailureException. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[
https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819403#comment-16819403
]
Íñigo Goiri commented on HDFS-14117:
Thanks [~daryn] for the feedback.
There are two parts to this JIRA.
The first one is to improve the way rename works by taking into account
incompatible subclusters.
The second one is the trash itself.
We can do the first one in a separate JIRA as it is the more fundamental one.
The second one, it's isolated to the createTrash function.
I tend to agree that the ideal scenario would be to provision the Trash before
any of this.
However, in default setups, this will fail.
Maybe make the creation optional?
Note that this is all based on the TRASH flag and there are no hard-coded paths.
RBF does not support EZ at this point so I haven't been able to test that.
> RBF: We can only delete the files or dirs of one subcluster in a cluster with
> multiple subclusters when trash is enabled
>
>
> Key: HDFS-14117
> URL: https://issues.apache.org/jira/browse/HDFS-14117
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: venkata ramkumar
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14117-HDFS-13891.001.patch,
> HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch,
> HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch,
> HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch,
> HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch,
> HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch,
> HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch,
> HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch,
> HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch,
> HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch,
> HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch,
> HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch
>
>
> When we delete files or dirs in hdfs, it will move the deleted files or dirs
> to trash by default.
> But in the global path we can only mount one trash dir /user. So we mount
> trash dir /user of the subcluster ns1 to the global path /user. Then we can
> delete files or dirs of ns1, but when we delete the files or dirs of another
> subcluser, such as hacluster, it will be failed.
> h1. Mount Table
> ||Global path||Target nameservice||Target path||Order||Read
> only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created||
> |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42|
> |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05|
> |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota:
> -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20|
> commands:
> {noformat}
> 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/.
> 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd
> 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/.
> 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd
> 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm
> /tmp/mapred.cmd
> 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination
> parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found.
> 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd
> 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved:
> 'hdfs://router/test/hdfs.cmd' to trash at:
> hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1269) [Ozone Upgrade] Add the ability to read block information from Namenode.
[ https://issues.apache.org/jira/browse/HDDS-1269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad reassigned HDDS-1269: -- Assignee: Mohammad > [Ozone Upgrade] Add the ability to read block information from Namenode. > > > Key: HDDS-1269 > URL: https://issues.apache.org/jira/browse/HDDS-1269 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: upgrade >Reporter: Anu Engineer >Assignee: Mohammad >Priority: Major > > When upgrading an HDFS cluster, the user can choose to upgrade with zero data > moves. However, it is also a good point to evaluate what-if? conditions and > make optimal decisions for both HDFS and Ozone. First step in this process is > to learn information about HDFS, FSImage provides then Namespace information, > this JIRA will provide the blocks and data nodes part of HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1268) [Ozone Upgrade] Add ability to read the FSImage from Namenode.
[ https://issues.apache.org/jira/browse/HDDS-1268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad reassigned HDDS-1268: -- Assignee: Mohammad > [Ozone Upgrade] Add ability to read the FSImage from Namenode. > -- > > Key: HDDS-1268 > URL: https://issues.apache.org/jira/browse/HDDS-1268 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: upgrade >Reporter: Anu Engineer >Assignee: Mohammad >Priority: Major > > Ozone upgrade planner should be able to show the current state of the HDFS > cluster, including paths, EC and TDE. This will allow user to choose what > parts of the cluster to upgrade to ozone and what should be the new path > name. In order to do this, Ozone upgrade planner should support the ability > to connect to name node, read FSImage and process it off-line. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[
https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819378#comment-16819378
]
Hadoop QA commented on HDFS-14117:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
5s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 11s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 74m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.contract.router.TestRouterHDFSContractRename |
| | hadoop.fs.contract.router.TestRouterHDFSContractRenameSecure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14117 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966115/HDFS-14117-HDFS-13891.020.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux fb4d86d5d399 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / bd3161e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26648/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26648/testReport/ |
| Max. process+thread count | 1004 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console
[jira] [Updated] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-10477:
---
Resolution: Fixed
Fix Version/s: 2.8.6
Status: Resolved (was: Patch Available)
Pushed to branch-2.8 Thanks [~starphin] for the help!
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Fix For: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3
>
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.004.patch, HDFS-10477.005.patch, HDFS-10477.006.patch,
> HDFS-10477.007.patch, HDFS-10477.branch-2.8.patch, HDFS-10477.branch-2.patch,
> HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819361#comment-16819361
]
Wei-Chiu Chuang commented on HDFS-10477:
Failures doesn't reproduce for me. Will commit now.
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Fix For: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 2.9.3, 3.1.3
>
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.004.patch, HDFS-10477.005.patch, HDFS-10477.006.patch,
> HDFS-10477.007.patch, HDFS-10477.branch-2.8.patch, HDFS-10477.branch-2.patch,
> HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
>
[jira] [Work logged] (HDDS-1441) Remove usage of getRetryFailureException
[ https://issues.apache.org/jira/browse/HDDS-1441?focusedWorklogId=228591=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228591 ] ASF GitHub Bot logged work on HDDS-1441: Author: ASF GitHub Bot Created on: 16/Apr/19 18:02 Start Date: 16/Apr/19 18:02 Worklog Time Spent: 10m Work Description: bshashikant commented on issue #745: HDDS-1441. Remove usage of getRetryFailureException. (swagle) URL: https://github.com/apache/hadoop/pull/745#issuecomment-483781495 I just verified that with Ratis-518, the exception field in RaftClientReply is already exposed. We can just wrap the exception inside RaftClientReply to completionException instead of checking for any specific exceptions. To acheive this, Ratis version needs to be updated inside Ozone. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228591) Time Spent: 40m (was: 0.5h) > Remove usage of getRetryFailureException > > > Key: HDDS-1441 > URL: https://issues.apache.org/jira/browse/HDDS-1441 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: Siddharth Wagle >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Per [~szetszwo]'s comment on RATIS-518, we can remove the usage of > getRetryFailureException. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[
https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819336#comment-16819336
]
Daryn Sharp commented on HDFS-14117:
It's a big patch and caught my eye because we are adding support for an
alternate hdfs trash path and this will of course complicate everything.
Anyway, the RBF is attempting to be too smart. It should know nothing about
the trash. The predecessor to httpfs (hdfsproxy) had all kinds of issues
because it tried to be smart instead of proxying the call through as-is. Very
subtle race conditions cropped up that were hard to debug.
Why not require the overlapping mount points to have a trash dir and then do
the fan-out rename?
Is this going to play nicely with EZ trashes?
> RBF: We can only delete the files or dirs of one subcluster in a cluster with
> multiple subclusters when trash is enabled
>
>
> Key: HDFS-14117
> URL: https://issues.apache.org/jira/browse/HDFS-14117
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: venkata ramkumar
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14117-HDFS-13891.001.patch,
> HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch,
> HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch,
> HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch,
> HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch,
> HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch,
> HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch,
> HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch,
> HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch,
> HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch,
> HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch,
> HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch
>
>
> When we delete files or dirs in hdfs, it will move the deleted files or dirs
> to trash by default.
> But in the global path we can only mount one trash dir /user. So we mount
> trash dir /user of the subcluster ns1 to the global path /user. Then we can
> delete files or dirs of ns1, but when we delete the files or dirs of another
> subcluser, such as hacluster, it will be failed.
> h1. Mount Table
> ||Global path||Target nameservice||Target path||Order||Read
> only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created||
> |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42|
> |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05|
> |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota:
> -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20|
> commands:
> {noformat}
> 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/.
> 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd
> 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/.
> 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd
> 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm
> /tmp/mapred.cmd
> 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination
> parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found.
> 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd
> 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved:
> 'hdfs://router/test/hdfs.cmd' to trash at:
> hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14418) Remove redundant super user priveledge checks from namenode.
[ https://issues.apache.org/jira/browse/HDFS-14418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819323#comment-16819323 ] Hudson commented on HDFS-14418: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16420 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16420/]) HDFS-14418. Remove redundant super user priveledge checks from namenode. (inigoiri: rev be6c8014e66be919388269b70cb2966c35b8c578) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java > Remove redundant super user priveledge checks from namenode. > > > Key: HDFS-14418 > URL: https://issues.apache.org/jira/browse/HDFS-14418 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14418-01.patch, HDFS-14418-02.patch > > > There are couple of methods that unnecessarily double checks super user > privileged at namenode, which can reduced to single. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14418) Remove redundant super user priveledge checks from namenode.
[ https://issues.apache.org/jira/browse/HDFS-14418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819322#comment-16819322 ] Íñigo Goiri commented on HDFS-14418: Thanks [~ayushtkn] for the patch and [~adam.antal] and [~daryn] for reviewing. Committed to trunk. > Remove redundant super user priveledge checks from namenode. > > > Key: HDFS-14418 > URL: https://issues.apache.org/jira/browse/HDFS-14418 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14418-01.patch, HDFS-14418-02.patch > > > There are couple of methods that unnecessarily double checks super user > privileged at namenode, which can reduced to single. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14418) Remove redundant super user priveledge checks from namenode.
[ https://issues.apache.org/jira/browse/HDFS-14418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14418: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.3.0 Status: Resolved (was: Patch Available) > Remove redundant super user priveledge checks from namenode. > > > Key: HDFS-14418 > URL: https://issues.apache.org/jira/browse/HDFS-14418 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14418-01.patch, HDFS-14418-02.patch > > > There are couple of methods that unnecessarily double checks super user > privileged at namenode, which can reduced to single. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14418) Remove redundant super user priveledge checks from namenode.
[ https://issues.apache.org/jira/browse/HDFS-14418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819317#comment-16819317 ] Íñigo Goiri commented on HDFS-14418: +1 on [^HDFS-14418-02.patch]. Committing shortly. > Remove redundant super user priveledge checks from namenode. > > > Key: HDFS-14418 > URL: https://issues.apache.org/jira/browse/HDFS-14418 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14418-01.patch, HDFS-14418-02.patch > > > There are couple of methods that unnecessarily double checks super user > privileged at namenode, which can reduced to single. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8631) WebHDFS : Support get/setQuota
[
https://issues.apache.org/jira/browse/HDFS-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819312#comment-16819312
]
Hadoop QA commented on HDFS-8631:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HDFS-8631 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-8631 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860576/HDFS-8631-006.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26649/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> WebHDFS : Support get/setQuota
> --
>
> Key: HDFS-8631
> URL: https://issues.apache.org/jira/browse/HDFS-8631
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.7.2
>Reporter: nijel
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-8631-001.patch, HDFS-8631-002.patch,
> HDFS-8631-003.patch, HDFS-8631-004.patch, HDFS-8631-005.patch,
> HDFS-8631-006.patch
>
>
> User is able do quota management from filesystem object. Same operation can
> be allowed trough REST API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14430) RBF: Fix MockNamenode bug about mocking RPC getListing and mkdir
[ https://issues.apache.org/jira/browse/HDFS-14430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819311#comment-16819311 ] Íñigo Goiri commented on HDFS-14430: [~hexiaoqiao] please take a look at the latest patch in HDFS-14117. This includes most of the changes you are considering here. Do you mind taking those here? BTW, for the changes, there should be a unit test that fails without them. > RBF: Fix MockNamenode bug about mocking RPC getListing and mkdir > > > Key: HDFS-14430 > URL: https://issues.apache.org/jira/browse/HDFS-14430 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: He Xiaoqiao >Assignee: He Xiaoqiao >Priority: Major > Attachments: HDFS-14430-HDFS-13891.001.patch > > > Some unexpected result when invoke mocking #getListing and #mkdirs in current > MockNamenode implement. > * for mock mkdirs, we do not check if parent directory exists. > * for mock getListing, some child dirs/files are not listing. > It may be cause some unexpected result and cause some unit test fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8631) WebHDFS : Support get/setQuota
[ https://issues.apache.org/jira/browse/HDFS-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819302#comment-16819302 ] Íñigo Goiri commented on HDFS-8631: --- Is there anybody working on this? [^HDFS-8631-006.patch] looks pretty much it. >From the discussion the issue seems HDFS vs FS. To avoid the controversy, I would just do the changes in the Namenode and we can discuss the WebHdfsAdmin vs WebHdfsFileSystem issue in a follow up. Then I would add the documentation to WebHDFS. Thoughts? > WebHDFS : Support get/setQuota > -- > > Key: HDFS-8631 > URL: https://issues.apache.org/jira/browse/HDFS-8631 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.2 >Reporter: nijel >Assignee: Surendra Singh Lilhore >Priority: Major > Attachments: HDFS-8631-001.patch, HDFS-8631-002.patch, > HDFS-8631-003.patch, HDFS-8631-004.patch, HDFS-8631-005.patch, > HDFS-8631-006.patch > > > User is able do quota management from filesystem object. Same operation can > be allowed trough REST API. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1432) Ozone client list command truncates response without any indication
[
https://issues.apache.org/jira/browse/HDDS-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819282#comment-16819282
]
Arpit Agarwal commented on HDDS-1432:
-
+1 for the patch. However the pull request looks empty.
> Ozone client list command truncates response without any indication
> ---
>
> Key: HDDS-1432
> URL: https://issues.apache.org/jira/browse/HDDS-1432
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Affects Versions: 0.5.0
>Reporter: Siddharth Wagle
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Attachments: HDDS-1432.01.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Trying to list all keys in a bucket gives truncated results unless the limit
> override is provided as such:
> {code}
> bin/ozone --config /etc/ozone/conf sh key list -l 5000 /vol-test-1/bucket1/
> {code}
> The proposal is to provide a warning about possible truncation of results and
> the override option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[
https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819277#comment-16819277
]
Íñigo Goiri commented on HDFS-14117:
Thanks [~ayushtkn] for the feedback.
I posted [^HDFS-14117-HDFS-13891.020.patch] with the approach that checks all
files.
> RBF: We can only delete the files or dirs of one subcluster in a cluster with
> multiple subclusters when trash is enabled
>
>
> Key: HDFS-14117
> URL: https://issues.apache.org/jira/browse/HDFS-14117
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: venkata ramkumar
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14117-HDFS-13891.001.patch,
> HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch,
> HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch,
> HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch,
> HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch,
> HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch,
> HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch,
> HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch,
> HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch,
> HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch,
> HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch,
> HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch
>
>
> When we delete files or dirs in hdfs, it will move the deleted files or dirs
> to trash by default.
> But in the global path we can only mount one trash dir /user. So we mount
> trash dir /user of the subcluster ns1 to the global path /user. Then we can
> delete files or dirs of ns1, but when we delete the files or dirs of another
> subcluser, such as hacluster, it will be failed.
> h1. Mount Table
> ||Global path||Target nameservice||Target path||Order||Read
> only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created||
> |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42|
> |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05|
> |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota:
> -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20|
> commands:
> {noformat}
> 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/.
> 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd
> 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/.
> 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd
> 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm
> /tmp/mapred.cmd
> 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination
> parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found.
> 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd
> 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved:
> 'hdfs://router/test/hdfs.cmd' to trash at:
> hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[
https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-14117:
---
Attachment: HDFS-14117-HDFS-13891.020.patch
> RBF: We can only delete the files or dirs of one subcluster in a cluster with
> multiple subclusters when trash is enabled
>
>
> Key: HDFS-14117
> URL: https://issues.apache.org/jira/browse/HDFS-14117
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: venkata ramkumar
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14117-HDFS-13891.001.patch,
> HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch,
> HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch,
> HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch,
> HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch,
> HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch,
> HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch,
> HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch,
> HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch,
> HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch,
> HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch,
> HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch
>
>
> When we delete files or dirs in hdfs, it will move the deleted files or dirs
> to trash by default.
> But in the global path we can only mount one trash dir /user. So we mount
> trash dir /user of the subcluster ns1 to the global path /user. Then we can
> delete files or dirs of ns1, but when we delete the files or dirs of another
> subcluser, such as hacluster, it will be failed.
> h1. Mount Table
> ||Global path||Target nameservice||Target path||Order||Read
> only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created||
> |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42|
> |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05|
> |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota:
> -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20|
> commands:
> {noformat}
> 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/.
> 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd
> 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/.
> 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd
> 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm
> /tmp/mapred.cmd
> 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination
> parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found.
> 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd
> 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved:
> 'hdfs://router/test/hdfs.cmd' to trash at:
> hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819252#comment-16819252
]
Daryn Sharp commented on HDFS-14390:
Ok, so the kerberos annotation is effectively the only change. You should make
a call via the fs instance, ex. getServerDefaults, to ensure that it can
actually connect and authenticate. The mini-cluster has probably already used
the fs but I wouldn't rely on it.
I don't think the {{FSTreeWalk}} change is a good idea. Is it actually needed?
The login user is the current user unless an explicit doAs another identity
has been done in which it was probably done for good reason. Why add a latent
surprise that second-guesses the caller and reverts back to the login user?
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch,
> HDFS-14390.003.patch, HDFS-14390.004.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819240#comment-16819240
]
Hadoop QA commented on HDFS-10477:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2.8 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
13s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} branch-2.8 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 145 unchanged - 1 fixed = 145 total (was 146) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}198m 52s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 2m
32s{color} | {color:red} The patch generated 197 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}239m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Unreaped Processes | hadoop-hdfs:50 |
| Failed junit tests | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.server.namenode.TestNameNodeRecovery |
| | hadoop.hdfs.server.namenode.TestNNThroughputBenchmark |
| | hadoop.hdfs.server.namenode.TestNameNodeRpcServer |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| | hadoop.hdfs.TestSetrepDecreasing |
| | hadoop.hdfs.server.namenode.TestFSNamesystem |
| | hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant |
| | hadoop.hdfs.TestFSInputChecker |
| | hadoop.hdfs.server.namenode.TestTransferFsImage |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.server.namenode.TestINodeAttributeProvider |
| | hadoop.hdfs.server.namenode.TestFSImageWithSnapshot |
| | hadoop.hdfs.TestFsShellPermission |
| | hadoop.hdfs.server.namenode.TestFSNamesystemLock |
| | hadoop.hdfs.TestEncryptionZonesWithHA |
| | hadoop.hdfs.TestDFSRename |
| Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
| | org.apache.hadoop.hdfs.TestFileCreationEmpty |
| | org.apache.hadoop.hdfs.TestDatanodeRegistration |
| | org.apache.hadoop.hdfs.server.namenode.TestFileContextAcl |
| | org.apache.hadoop.hdfs.TestBlocksScheduledCounter |
| | org.apache.hadoop.hdfs.TestDFSClientFailover |
| | org.apache.hadoop.hdfs.TestSetrepIncreasing |
| | org.apache.hadoop.hdfs.server.namenode.TestINodeFile |
| | org.apache.hadoop.hdfs.TestDatanodeDeath |
| | org.apache.hadoop.hdfs.TestDFSClientRetries |
| |
[jira] [Commented] (HDFS-14430) RBF: Fix MockNamenode bug about mocking RPC getListing and mkdir
[
https://issues.apache.org/jira/browse/HDFS-14430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819223#comment-16819223
]
Hadoop QA commented on HDFS-14430:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 39m
10s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 15s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch
generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 58s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 22m 31s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRouterFaultTolerant |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14430 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966096/HDFS-14430-HDFS-13891.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d32a8d378346 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / bd3161e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26647/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26647/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| Test Results |
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819200#comment-16819200
]
Ashvin commented on HDFS-14390:
---
Hi [~daryn] I wanted to follow up on this issue. I have uploaded a new patch,
[^HDFS-14390.004.patch]. As discussed earlier, the patch includes just the
changes required to enable secure connection to the {{AliasMap}} server and
fixes the image generation tool used for provided storage, see {{FSTreeWalk}}.
Auth related changes will be part of a new PR.
Summary: When authentication method is {{Kerberos}}, a client (DN/NN in this
case) invokes {{SaslRpcClient.getServerPrincipal}} to setup a secure
connection. If {{Provided storage}} is also enabled, the {{getServerPrincipal}}
method tries to find server principal for the {{AliasMap}} protocol. It was
missing earlier. This change, see {{AliasMapProtocolPB}}, provides the server
principal. No other components or paths will be impacted by this change.
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch,
> HDFS-14390.003.patch, HDFS-14390.004.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13596) NN restart fails after RollingUpgrade from 2.x to 3.x
[
https://issues.apache.org/jira/browse/HDFS-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819198#comment-16819198
]
Daryn Sharp commented on HDFS-13596:
Consider whether it would be simpler for {{FSEditLog#logOpenFile}} to put
something (layout, boolean, etc) in the {{AddOp}} to indicate whether the EC
field should be written. Then we don't need to track and burrow the layout
through the edit log, stream impls, double buf, etc.
If not, might consider adding this to significantly reduce patch size:
{code}
public void writeFields(DataOutputStream out, int logVersion) throws
IOException {
writeFields(out);
}
{code}
Test case should use a Feature enum prior to EC instead of hardcoding the
number. I only skimmed the patch (need to look at it more) but not sure the
test case provides enough coverage. It appears to assume the ability to read
the edit means it was written "correctly" w/o the EC field. If yes, that's a
really bad assumption.
> NN restart fails after RollingUpgrade from 2.x to 3.x
> -
>
> Key: HDFS-13596
> URL: https://issues.apache.org/jira/browse/HDFS-13596
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Hanisha Koneru
>Assignee: Fei Hui
>Priority: Critical
> Attachments: HDFS-13596.001.patch, HDFS-13596.002.patch,
> HDFS-13596.003.patch, HDFS-13596.004.patch, HDFS-13596.005.patch,
> HDFS-13596.006.patch, HDFS-13596.007.patch
>
>
> After rollingUpgrade NN from 2.x and 3.x, if the NN is restarted, it fails
> while replaying edit logs.
> * After NN is started with rollingUpgrade, the layoutVersion written to
> editLogs (before finalizing the upgrade) is the pre-upgrade layout version
> (so as to support downgrade).
> * When writing transactions to log, NN writes as per the current layout
> version. In 3.x, erasureCoding bits are added to the editLog transactions.
> * So any edit log written after the upgrade and before finalizing the
> upgrade will have the old layout version but the new format of transactions.
> * When NN is restarted and the edit logs are replayed, the NN reads the old
> layout version from the editLog file. When parsing the transactions, it
> assumes that the transactions are also from the previous layout and hence
> skips parsing the erasureCoding bits.
> * This cascades into reading the wrong set of bits for other fields and
> leads to NN shutting down.
> Sample error output:
> {code:java}
> java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected
> length 16
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:74)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:86)
> at
> org.apache.hadoop.ipc.RetryCache$CacheEntryWithPayload.(RetryCache.java:163)
> at
> org.apache.hadoop.ipc.RetryCache.addCacheEntryWithPayload(RetryCache.java:322)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheEntryWithPayload(FSNamesystem.java:960)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:397)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:249)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:888)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1086)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:937)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:910)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710)
> 2018-05-17 19:10:06,522 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: java.lang.IllegalStateException: Cannot skip to less
> than the current value (=16389), where newValue=16388
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.resetLastInodeId(FSDirectory.java:1945)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:298)
> at
>
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[
https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819155#comment-16819155
]
Ayush Saxena commented on HDFS-14117:
-
Thanx [~elgoiri] for the briefings.
Tough choice to make, when both solutions are technically correct. BEST vs BEST
leads to one and WORST vs WORST leads to another.
Some point I feel that Second case hitting the best case may be more but as we
can't guarantee it, so we need to consider an average case considering both
positive and negative cases happen at almost same rate, So I won't make a call
based on number of RPC here.
Preventing writeLock shall be then a better choice vs saving RPC's. If you too
are good with it, I am happy going ahead with it.
> RBF: We can only delete the files or dirs of one subcluster in a cluster with
> multiple subclusters when trash is enabled
>
>
> Key: HDFS-14117
> URL: https://issues.apache.org/jira/browse/HDFS-14117
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: venkata ramkumar
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14117-HDFS-13891.001.patch,
> HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch,
> HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch,
> HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch,
> HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch,
> HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch,
> HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch,
> HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch,
> HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch,
> HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch,
> HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch,
> HDFS-14117.004.patch, HDFS-14117.005.patch
>
>
> When we delete files or dirs in hdfs, it will move the deleted files or dirs
> to trash by default.
> But in the global path we can only mount one trash dir /user. So we mount
> trash dir /user of the subcluster ns1 to the global path /user. Then we can
> delete files or dirs of ns1, but when we delete the files or dirs of another
> subcluser, such as hacluster, it will be failed.
> h1. Mount Table
> ||Global path||Target nameservice||Target path||Order||Read
> only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created||
> |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42|
> |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05|
> |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota:
> -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20|
> commands:
> {noformat}
> 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/.
> 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd
> 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/.
> 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd
> 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm
> /tmp/mapred.cmd
> 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination
> parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found.
> 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd
> 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved:
> 'hdfs://router/test/hdfs.cmd' to trash at:
> hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14403) Cost-Based RPC FairCallQueue
[ https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819133#comment-16819133 ] Erik Krogen commented on HDFS-14403: Sure, enjoy your vacation with peace of mind that we won't sneak anything by you :) > Cost-Based RPC FairCallQueue > > > Key: HDFS-14403 > URL: https://issues.apache.org/jira/browse/HDFS-14403 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ipc, namenode >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Labels: qos, rpc > Attachments: CostBasedFairCallQueueDesign_v0.pdf, > HDFS-14403.001.patch, HDFS-14403.002.patch, HDFS-14403.003.patch, > HDFS-14403.branch-2.8.patch > > > HADOOP-15016 initially described extensions to the Hadoop FairCallQueue > encompassing both cost-based analysis of incoming RPCs, as well as support > for reservations of RPC capacity for system/platform users. This JIRA intends > to track the former, as HADOOP-15016 was repurposed to more specifically > focus on the reservation portion of the work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14403) Cost-Based RPC FairCallQueue
[ https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819126#comment-16819126 ] Daryn Sharp commented on HDFS-14403: Fantastic. I'd like a chance to review the patch so please remember I'm on vacation next week. > Cost-Based RPC FairCallQueue > > > Key: HDFS-14403 > URL: https://issues.apache.org/jira/browse/HDFS-14403 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ipc, namenode >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Labels: qos, rpc > Attachments: CostBasedFairCallQueueDesign_v0.pdf, > HDFS-14403.001.patch, HDFS-14403.002.patch, HDFS-14403.003.patch, > HDFS-14403.branch-2.8.patch > > > HADOOP-15016 initially described extensions to the Hadoop FairCallQueue > encompassing both cost-based analysis of incoming RPCs, as well as support > for reservations of RPC capacity for system/platform users. This JIRA intends > to track the former, as HADOOP-15016 was repurposed to more specifically > focus on the reservation portion of the work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14430) RBF: Fix MockNamenode bug about mocking RPC getListing and mkdir
[
https://issues.apache.org/jira/browse/HDFS-14430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819107#comment-16819107
]
He Xiaoqiao commented on HDFS-14430:
Thanks [~ayushtkn] for your quick response, I just check interface using
MockNamenode locally, but some result is not my expectation. For instance
using MockNamenode,
1. mkdir '/user', '/user/hive/warehouse', '/user/hadoop/test';
2. then get null when invoke getListing of '/user'.
I expect the correct result may be {'/user/hive', '/user/hadoop'}.
> RBF: Fix MockNamenode bug about mocking RPC getListing and mkdir
>
>
> Key: HDFS-14430
> URL: https://issues.apache.org/jira/browse/HDFS-14430
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-14430-HDFS-13891.001.patch
>
>
> Some unexpected result when invoke mocking #getListing and #mkdirs in current
> MockNamenode implement.
> * for mock mkdirs, we do not check if parent directory exists.
> * for mock getListing, some child dirs/files are not listing.
> It may be cause some unexpected result and cause some unit test fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
