[jira] [Resolved] (HDFS-14551) NN throws NPE if downgrade it during rolling upgrade from 3.x to 2.x
[
https://issues.apache.org/jira/browse/HDFS-14551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuxuan Wang resolved HDFS-14551.
Resolution: Duplicate
> NN throws NPE if downgrade it during rolling upgrade from 3.x to 2.x
>
>
> Key: HDFS-14551
> URL: https://issues.apache.org/jira/browse/HDFS-14551
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Yuxuan Wang
>Priority: Blocker
>
> We can downgrade NN during roling upgrade (runned "hdfs dfsadmin
> -rollingUpgrade prepare) with HDFS-8432 involved. But with HDFS-14172 if the
> image has any unrecogized section, it will throw IOException at
> {code:title=FSImageFormatProtobuf.java|borderStyle=solid}
> private void loadInternal(..) {
> ..
> String n = s.getName();
> SectionName sectionName = SectionName.fromString(n);
> if (sectionName == null) {
> throw new IOException("Unrecognized section " + n);
> }
> ..
> }
> {code}
> and throw NPE on Hadoop 2.x
> {code:title=FSImageFormatProtobuf.java|borderStyle=solid}
> private void loadInternal(..) {
> ..
> String n = s.getName();
> switch (sectionName)
> ..
> }
> {code}
> When we downgrade NN from 3.x to 2.x, NN may load the image saved by 3.x NN.
> Then the lack of {{SectionName.ERASURE_CODING}} can break 2.x NN.
> We should just skip the unrecogized section instead of throwing exception.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14311) multi-threading conflict at layoutVersion when loading block pool storage
[
https://issues.apache.org/jira/browse/HDFS-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910187#comment-16910187
]
Surendra Singh Lilhore commented on HDFS-14311:
---
Thanks [~caiyicong] for reporting this issue.
We got the same issue in our cluster and it got fixed with this patch. I don't
think it is easy to reproduce in unit test.
> multi-threading conflict at layoutVersion when loading block pool storage
> -
>
> Key: HDFS-14311
> URL: https://issues.apache.org/jira/browse/HDFS-14311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rolling upgrades
>Affects Versions: 2.9.2
>Reporter: Yicong Cai
>Assignee: Yicong Cai
>Priority: Major
> Attachments: HDFS-14311.1.patch
>
>
> When DataNode upgrade from 2.7.3 to 2.9.2, there is a conflict at
> StorageInfo.layoutVersion in loading block pool storage process.
> It will cause this exception:
>
> {panel:title=exceptions}
> 2019-02-15 10:18:01,357 [13783] - INFO [Thread-33:BlockPoolSliceStorage@395]
> - Restored 36974 block files from trash before the layout upgrade. These
> blocks will be moved to the previous directory during the upgrade
> 2019-02-15 10:18:01,358 [13784] - WARN [Thread-33:BlockPoolSliceStorage@226]
> - Failed to analyze storage directories for block pool
> BP-1216718839-10.120.232.23-1548736842023
> java.io.IOException: Datanode state: LV = -57 CTime = 0 is newer than the
> namespace state: LV = -63 CTime = 0
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.doTransition(BlockPoolSliceStorage.java:406)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadStorageDirectory(BlockPoolSliceStorage.java:177)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadBpStorageDirectories(BlockPoolSliceStorage.java:221)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.recoverTransitionRead(BlockPoolSliceStorage.java:250)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.loadBlockPoolSliceStorage(DataStorage.java:460)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:390)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:556)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1649)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1610)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:388)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:280)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:816)
> at java.lang.Thread.run(Thread.java:748)
> 2019-02-15 10:18:01,358 [13784] - WARN [Thread-33:DataStorage@472] - Failed
> to add storage directory [DISK]file:/mnt/dfs/2/hadoop/hdfs/data/ for block
> pool BP-1216718839-10.120.232.23-1548736842023
> java.io.IOException: Datanode state: LV = -57 CTime = 0 is newer than the
> namespace state: LV = -63 CTime = 0
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.doTransition(BlockPoolSliceStorage.java:406)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadStorageDirectory(BlockPoolSliceStorage.java:177)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadBpStorageDirectories(BlockPoolSliceStorage.java:221)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.recoverTransitionRead(BlockPoolSliceStorage.java:250)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.loadBlockPoolSliceStorage(DataStorage.java:460)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:390)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:556)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1649)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1610)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:388)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:280)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:816)
> at java.lang.Thread.run(Thread.java:748)
> {panel}
>
> root cause:
> BlockPoolSliceStorage instance is shared for all storage locations recover
> transition. In BlockPoolSliceStorage.doTransition, it will read the old
> layoutVersion from local storage, compare with current DataNode version, then
> do upgrade. In doUpgrade, add the
[jira] [Work logged] (HDDS-1610) applyTransaction failure should not be lost on restart
[ https://issues.apache.org/jira/browse/HDDS-1610?focusedWorklogId=297199=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297199 ] ASF GitHub Bot logged work on HDDS-1610: Author: ASF GitHub Bot Created on: 19/Aug/19 15:16 Start Date: 19/Aug/19 15:16 Worklog Time Spent: 10m Work Description: bshashikant commented on issue #1226: HDDS-1610. applyTransaction failure should not be lost on restart. URL: https://github.com/apache/hadoop/pull/1226#issuecomment-522623256 The test failures are not related to the patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297199) Time Spent: 5h 50m (was: 5h 40m) > applyTransaction failure should not be lost on restart > -- > > Key: HDDS-1610 > URL: https://issues.apache.org/jira/browse/HDDS-1610 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Blocker > Labels: pull-request-available > Time Spent: 5h 50m > Remaining Estimate: 0h > > If the applyTransaction fails in the containerStateMachine, then the > container should not accept new writes on restart,. > This can occur if > # chunk write applyTransaction fails > # container state update to UNHEALTHY also fails > # Ratis snapshot is taken > # Node restarts > # container accepts new transactions -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14476) lock too long when fix inconsistent blocks between disk and in-memory
[
https://issues.apache.org/jira/browse/HDFS-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14476:
---
Fix Version/s: 3.1.3
3.2.1
Resolution: Fixed
Status: Resolved (was: Patch Available)
Failed test doesn't look related. I pushed the patch to branch-3.2 and
branch-3.1.
Awesome, it's always great to see a patch get deployed & tested before posted
upstream.
> lock too long when fix inconsistent blocks between disk and in-memory
> -
>
> Key: HDFS-14476
> URL: https://issues.apache.org/jira/browse/HDFS-14476
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.6.0, 2.7.0, 3.0.3
>Reporter: Sean Chow
>Assignee: Sean Chow
>Priority: Major
> Fix For: 2.10.0, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3
>
> Attachments: HDFS-14476-branch-2.01.patch, HDFS-14476.00.patch,
> HDFS-14476.002.patch, HDFS-14476.01.patch, HDFS-14476.branch-3.2.001.patch,
> datanode-with-patch-14476.png
>
>
> When directoryScanner have the results of differences between disk and
> in-memory blocks. it will try to run {{checkAndUpdate}} to fix it. However
> {{FsDatasetImpl.checkAndUpdate}} is a synchronized call
> As I have about 6millions blocks for every datanodes and every 6hours' scan
> will have about 25000 abnormal blocks to fix. That leads to a long lock
> holding FsDatasetImpl object.
> let's assume every block need 10ms to fix(because of latency of SAS disk),
> that will cost 250 seconds to finish. That means all reads and writes will be
> blocked for 3mins for that datanode.
>
> {code:java}
> 2019-05-06 08:06:51,704 INFO
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool
> BP-1644920766-10.223.143.220-1450099987967 Total blocks: 6850197, missing
> metadata files:23574, missing block files:23574, missing blocks in
> memory:47625, mismatched blocks:0
> ...
> 2019-05-06 08:16:41,625 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 588402ms to process 1 commands from NN
> {code}
> Take long time to process command from nn because threads are blocked. And
> namenode will see long lastContact time for this datanode.
> Maybe this affect all hdfs versions.
> *how to fix:*
> just like process invalidate command from namenode with 1000 batch size, fix
> these abnormal block should be handled with batch too and sleep 2 seconds
> between the batch to allow normal reading/writing blocks.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1927) Consolidate add/remove Acl into OzoneAclUtil class
[ https://issues.apache.org/jira/browse/HDDS-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-1927: --- Target Version/s: 0.4.1 > Consolidate add/remove Acl into OzoneAclUtil class > -- > > Key: HDDS-1927 > URL: https://issues.apache.org/jira/browse/HDDS-1927 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > This Jira is created based on @xiaoyu comment on HDDS-1884 > Can we abstract these add/remove logic into common AclUtil class as we can > see similar logic in both bucket manager and key manager? For example, > public static boolean addAcl(List existingAcls, OzoneAcl newAcl) > public static boolean removeAcl(List existingAcls, OzoneAcl newAcl) > > But to do this, we need both OmKeyInfo and OMBucketInfo to use list of > OzoneAcl/OzoneAclInfo. > This Jira is to do that refactor, and also address above comment to move > common logic to AclUtils. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14714) RBF: implement getReplicatedBlockStats interface
[
https://issues.apache.org/jira/browse/HDFS-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910615#comment-16910615
]
Hadoop QA commented on HDFS-14714:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch
generated 2 new + 5 unchanged - 0 fixed = 7 total (was 5) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 45s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 0s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken |
| | hadoop.hdfs.server.federation.router.TestRouterWithSecureStartup |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14714 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977984/HDFS-14714.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e4a7841e80e7 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c765584 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/27565/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/27565/artifact/out/whitespace-eol.txt
[jira] [Updated] (HDFS-14665) HttpFS: LISTSTATUS response is missing HDFS-specific fields
[
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14665:
---
Status: Patch Available (was: Reopened)
> HttpFS: LISTSTATUS response is missing HDFS-specific fields
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Fix For: 3.3.0, 3.2.1
>
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
> }
> ]
> }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
> }
> ]
> }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (should only be none 0 for directories)
> "fileId"
> "storagePolicy"
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same
> underlying calls to compose the response.
> Root cause:
> [toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
> didn't serialize the HDFS-specific keys from FileStatus.
> Also may file another Jira to align the order of the keys in the responses.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14749) RBF: Isolation across multiple downstream hdfs clusters
[ https://issues.apache.org/jira/browse/HDFS-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CR Hota updated HDFS-14749: --- Summary: RBF: Isolation across multiple downstream hdfs clusters (was: Isolation across multiple downstream hdfs clusters) > RBF: Isolation across multiple downstream hdfs clusters > --- > > Key: HDFS-14749 > URL: https://issues.apache.org/jira/browse/HDFS-14749 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > > This parent Jira tracks all work done within the context of router isolation > across multiple downstream hdfs clusters. > # Phase 1 will be static allocation of resources. > # Phase 2 will introduce more dynamic approach/preemption based. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14729) Upgrade Bootstrap and jQuery versions used in HDFS UIs
[ https://issues.apache.org/jira/browse/HDFS-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vivek Ratnavel Subramanian updated HDFS-14729: -- Status: Patch Available (was: Open) > Upgrade Bootstrap and jQuery versions used in HDFS UIs > -- > > Key: HDFS-14729 > URL: https://issues.apache.org/jira/browse/HDFS-14729 > Project: Hadoop HDFS > Issue Type: Task > Components: ui >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Attachments: HDFS-14729.v1.patch > > > The current versions of bootstrap and jquery have multiple medium severity > CVEs reported till date and needs to be updated to the latest versions with > no reported CVEs. > > I suggest updating the following libraries: > ||Library||From version||To version|| > |Bootstrap|3.3.7|3.4.1| > |jQuery|3.3.1|3.4.1| -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14729) Upgrade Bootstrap and jQuery versions used in HDFS UIs
[ https://issues.apache.org/jira/browse/HDFS-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vivek Ratnavel Subramanian updated HDFS-14729: -- Status: Open (was: Patch Available) > Upgrade Bootstrap and jQuery versions used in HDFS UIs > -- > > Key: HDFS-14729 > URL: https://issues.apache.org/jira/browse/HDFS-14729 > Project: Hadoop HDFS > Issue Type: Task > Components: ui >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Attachments: HDFS-14729.v1.patch > > > The current versions of bootstrap and jquery have multiple medium severity > CVEs reported till date and needs to be updated to the latest versions with > no reported CVEs. > > I suggest updating the following libraries: > ||Library||From version||To version|| > |Bootstrap|3.3.7|3.4.1| > |jQuery|3.3.1|3.4.1| -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14749) Isolation across multiple downstream hdfs clusters
CR Hota created HDFS-14749: -- Summary: Isolation across multiple downstream hdfs clusters Key: HDFS-14749 URL: https://issues.apache.org/jira/browse/HDFS-14749 Project: Hadoop HDFS Issue Type: Sub-task Reporter: CR Hota Assignee: CR Hota This parent Jira tracks all work done within the context of router isolation across multiple downstream hdfs clusters. # Phase 1 will be static allocation of resources. # Phase 2 will introduce more dynamic approach/preemption based. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13709) Report bad block to NN when transfer block encounter EIO exception
[ https://issues.apache.org/jira/browse/HDFS-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-13709: --- Fix Version/s: 3.1.3 3.2.1 3.3.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed 005 to trunk, branch-3.2 and branch-3.1. There are conflicts cherry picking the commit into branch-2 and lower. [~zhangchen] if you are interested please provide a branch-2 patch. Thanks [~zhangchen] and [~sodonnell]! > Report bad block to NN when transfer block encounter EIO exception > -- > > Key: HDFS-13709 > URL: https://issues.apache.org/jira/browse/HDFS-13709 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Chen Zhang >Assignee: Chen Zhang >Priority: Major > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: HDFS-13709.002.patch, HDFS-13709.003.patch, > HDFS-13709.004.patch, HDFS-13709.005.patch, HDFS-13709.patch > > > In our online cluster, the BlockPoolSliceScanner is turned off, and sometimes > disk bad track may cause data loss. > For example, there are 3 replicas on 3 machines A/B/C, if a bad track occurs > on A's replica data, and someday B and C crushed at the same time, NN will > try to replicate data from A but failed, this block is corrupt now but no one > knows, because NN think there is at least 1 healthy replica and it keep > trying to replicate it. > When reading a replica which have data on bad track, OS will return an EIO > error, if DN reports the bad block as soon as it got an EIO, we can find > this case ASAP and try to avoid data loss -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12510) RBF: Add security to UI
[ https://issues.apache.org/jira/browse/HDFS-12510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910726#comment-16910726 ] CR Hota commented on HDFS-12510: Thanks [~elgoiri] > RBF: Add security to UI > --- > > Key: HDFS-12510 > URL: https://issues.apache.org/jira/browse/HDFS-12510 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Labels: RBF > > HDFS-12273 implemented the UI for Router Based Federation without security. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12510) RBF: Add security to UI
[ https://issues.apache.org/jira/browse/HDFS-12510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CR Hota resolved HDFS-12510. Resolution: Resolved > RBF: Add security to UI > --- > > Key: HDFS-12510 > URL: https://issues.apache.org/jira/browse/HDFS-12510 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Labels: RBF > > HDFS-12273 implemented the UI for Router Based Federation without security. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1972) Provide example ha proxy with multiple s3 servers back end.
[
https://issues.apache.org/jira/browse/HDDS-1972?focusedWorklogId=297389=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297389
]
ASF GitHub Bot logged work on HDDS-1972:
Author: ASF GitHub Bot
Created on: 19/Aug/19 20:21
Start Date: 19/Aug/19 20:21
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #1304: HDDS-1972.
Provide example ha proxy with multiple s3 servers back end.
URL: https://github.com/apache/hadoop/pull/1304#discussion_r315393614
##
File path:
hadoop-ozone/dist/src/main/compose/ozones3-haproxy/docker-compose.yaml
##
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ s3g:
+ image: haproxy:latest
+ volumes:
+ - ../..:/opt/hadoop
+ - ./haproxy-conf/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg
+ ports:
+ - 9878:9878
+ datanode:
+ image: apache/ozone-runner:${HADOOP_RUNNER_VERSION}
+ volumes:
+- ../..:/opt/hadoop
+ ports:
+- 9864
+ command: ["ozone","datanode"]
+ env_file:
+- ./docker-config
+ om:
+ image: apache/ozone-runner:${HADOOP_RUNNER_VERSION}
+ volumes:
+ - ../..:/opt/hadoop
+ ports:
+ - 9874:9874
+ environment:
+ ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
+ env_file:
+ - ./docker-config
+ command: ["ozone","om"]
+ scm:
+ image: apache/ozone-runner:${HADOOP_RUNNER_VERSION}
+ volumes:
+ - ../..:/opt/hadoop
+ ports:
+ - 9876:9876
+ env_file:
+ - ./docker-config
+ environment:
+ ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
+ command: ["ozone","scm"]
+ s3g1:
Review comment:
These can be collapsed into one entry, then we can use scale option. Not a
blocker to commit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297389)
Time Spent: 4h 10m (was: 4h)
> Provide example ha proxy with multiple s3 servers back end.
> ---
>
> Key: HDDS-1972
> URL: https://issues.apache.org/jira/browse/HDDS-1972
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> In this Jira, we shall provide docker-compose files where we start 3 s3
> gateway servers, and ha-proxy is used to load balance these S3 Gateway
> Servers.
>
> In this Jira, all are proxy configurations are hardcoded, we can make
> improvements to scale and automatically configure with environment variables
> as a future improvement. This is just a starter example.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14583) FileStatus#toString() will throw IllegalArgumentException
[
https://issues.apache.org/jira/browse/HDFS-14583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910725#comment-16910725
]
Íñigo Goiri commented on HDFS-14583:
I would make the test a little more precise.
Right now it just checks for one of the fields.
We should get something close to a full assertEquals.
> FileStatus#toString() will throw IllegalArgumentException
> -
>
> Key: HDFS-14583
> URL: https://issues.apache.org/jira/browse/HDFS-14583
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: xuzq
>Assignee: xuzq
>Priority: Major
> Labels: HDFS
> Attachments: HDFS-14583-trunk-0001.patch
>
>
> FileStatus#toString() will throw IllegalArgumentException, stack and error
> message like this:
> {code:java}
> java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:172)
> at org.apache.hadoop.fs.Path.(Path.java:184)
> at
> org.apache.hadoop.hdfs.protocol.HdfsLocatedFileStatus.getSymlink(HdfsLocatedFileStatus.java:117)
> at org.apache.hadoop.fs.FileStatus.toString(FileStatus.java:462)
> at
> org.apache.hadoop.hdfs.web.TestJsonUtil.testHdfsFileStatus(TestJsonUtil.java:123)
> {code}
> Test Code like this:
> {code:java}
> @Test
> public void testHdfsFileStatus() throws IOException {
> HdfsFileStatus hdfsFileStatus = new HdfsFileStatus.Builder()
> .replication(1)
> .blocksize(1024)
> .perm(new FsPermission((short) 777))
> .owner("owner")
> .group("group")
> .symlink(new byte[0])
> .path(new byte[0])
> .fileId(1010)
> .isdir(true)
> .build();
> System.out.println("HdfsFileStatus = " + hdfsFileStatus.toString());
> }{code}
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14396) Failed to load image from FSImageFile when downgrade from 3.x to 2.x
[
https://issues.apache.org/jira/browse/HDFS-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910678#comment-16910678
]
Wei-Chiu Chuang commented on HDFS-14396:
I just like to make sure I understand the behavior of this patch more carefully:
does it mean that until the upgrade is finalized, namenode can not persist
erasure coding information, and therefore a user should not start using erasure
coding until finalize is done?
If so, I think we should document this behavior somewhere (in the release note
or the "HDFS Rolling Upgrade" doc) as it is not so obvious. Thanks!
> Failed to load image from FSImageFile when downgrade from 3.x to 2.x
>
>
> Key: HDFS-14396
> URL: https://issues.apache.org/jira/browse/HDFS-14396
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-14396.001.patch, HDFS-14396.002.patch
>
>
> After fixing HDFS-13596, try to downgrade from 3.x to 2.x. But namenode can't
> start because exception occurs. The message follows
> {code:java}
> 2019-01-23 17:22:18,730 ERROR org.apache.hadoop.hdfs.server.namenode.FSImage:
> Failed to load image from
> FSImageFile(file=/data1/hadoopdata/hadoop-namenode/current/fsimage_0025310,
> cpktTxId=00
> 25310)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:179)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:885)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:869)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:742)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:673)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:290)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:998)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:612)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:672)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:839)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:823)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1517)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1583)
> 2019-01-23 17:22:19,023 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: Failed to load FSImage file, see error(s) above for more
> info.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:688)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:290)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:998)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:612)
> {code}
> This issue occurs because 3.x namenode saves image with EC fields during
> upgrade
> Try to fix it
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14090) RBF: Improved isolation for downstream name nodes. {Static}
[
https://issues.apache.org/jira/browse/HDFS-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
CR Hota updated HDFS-14090:
---
Parent: (was: HDFS-14603)
Issue Type: New Feature (was: Sub-task)
> RBF: Improved isolation for downstream name nodes. {Static}
> ---
>
> Key: HDFS-14090
> URL: https://issues.apache.org/jira/browse/HDFS-14090
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14090-HDFS-13891.001.patch,
> HDFS-14090-HDFS-13891.002.patch, HDFS-14090-HDFS-13891.003.patch,
> HDFS-14090-HDFS-13891.004.patch, HDFS-14090-HDFS-13891.005.patch,
> HDFS-14090.006.patch, HDFS-14090.007.patch, HDFS-14090.008.patch,
> HDFS-14090.009.patch, HDFS-14090.010.patch, RBF_ Isolation design.pdf
>
>
> Router is a gateway to underlying name nodes. Gateway architectures, should
> help minimize impact of clients connecting to healthy clusters vs unhealthy
> clusters.
> For example - If there are 2 name nodes downstream, and one of them is
> heavily loaded with calls spiking rpc queue times, due to back pressure the
> same with start reflecting on the router. As a result of this, clients
> connecting to healthy/faster name nodes will also slow down as same rpc queue
> is maintained for all calls at the router layer. Essentially the same IPC
> thread pool is used by router to connect to all name nodes.
> Currently router uses one single rpc queue for all calls. Lets discuss how we
> can change the architecture and add some throttling logic for
> unhealthy/slow/overloaded name nodes.
> One way could be to read from current call queue, immediately identify
> downstream name node and maintain a separate queue for each underlying name
> node. Another simpler way is to maintain some sort of rate limiter configured
> for each name node and let routers drop/reject/send error requests after
> certain threshold.
> This won’t be a simple change as router’s ‘Server’ layer would need redesign
> and implementation. Currently this layer is the same as name node.
> Opening this ticket to discuss, design and implement this feature.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1972) Provide example ha proxy with multiple s3 servers back end.
[ https://issues.apache.org/jira/browse/HDDS-1972?focusedWorklogId=297410=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297410 ] ASF GitHub Bot logged work on HDDS-1972: Author: ASF GitHub Bot Created on: 19/Aug/19 20:59 Start Date: 19/Aug/19 20:59 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #1304: HDDS-1972. Provide example ha proxy with multiple s3 servers back end. URL: https://github.com/apache/hadoop/pull/1304#issuecomment-522751627 Thank You @arp7 and @adoroszlai for the review. I will commit this to the trunk. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297410) Time Spent: 4.5h (was: 4h 20m) > Provide example ha proxy with multiple s3 servers back end. > --- > > Key: HDDS-1972 > URL: https://issues.apache.org/jira/browse/HDDS-1972 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 4.5h > Remaining Estimate: 0h > > In this Jira, we shall provide docker-compose files where we start 3 s3 > gateway servers, and ha-proxy is used to load balance these S3 Gateway > Servers. > > In this Jira, all are proxy configurations are hardcoded, we can make > improvements to scale and automatically configure with environment variables > as a future improvement. This is just a starter example. > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1972) Provide example ha proxy with multiple s3 servers back end.
[
https://issues.apache.org/jira/browse/HDDS-1972?focusedWorklogId=297409=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297409
]
ASF GitHub Bot logged work on HDDS-1972:
Author: ASF GitHub Bot
Created on: 19/Aug/19 20:59
Start Date: 19/Aug/19 20:59
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1304:
HDDS-1972. Provide example ha proxy with multiple s3 servers back end.
URL: https://github.com/apache/hadoop/pull/1304#discussion_r315409086
##
File path:
hadoop-ozone/dist/src/main/compose/ozones3-haproxy/docker-compose.yaml
##
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ s3g:
+ image: haproxy:latest
+ volumes:
+ - ../..:/opt/hadoop
+ - ./haproxy-conf/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg
+ ports:
+ - 9878:9878
+ datanode:
+ image: apache/ozone-runner:${HADOOP_RUNNER_VERSION}
+ volumes:
+- ../..:/opt/hadoop
+ ports:
+- 9864
+ command: ["ozone","datanode"]
+ env_file:
+- ./docker-config
+ om:
+ image: apache/ozone-runner:${HADOOP_RUNNER_VERSION}
+ volumes:
+ - ../..:/opt/hadoop
+ ports:
+ - 9874:9874
+ environment:
+ ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
+ env_file:
+ - ./docker-config
+ command: ["ozone","om"]
+ scm:
+ image: apache/ozone-runner:${HADOOP_RUNNER_VERSION}
+ volumes:
+ - ../..:/opt/hadoop
+ ports:
+ - 9876:9876
+ env_file:
+ - ./docker-config
+ environment:
+ ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
+ command: ["ozone","scm"]
+ s3g1:
Review comment:
Yes, we can do that, but we need to have ha-proxy image which can have some
configuration, where we can dynamically configure s3proxy and start/stop.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297409)
Time Spent: 4h 20m (was: 4h 10m)
> Provide example ha proxy with multiple s3 servers back end.
> ---
>
> Key: HDDS-1972
> URL: https://issues.apache.org/jira/browse/HDDS-1972
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> In this Jira, we shall provide docker-compose files where we start 3 s3
> gateway servers, and ha-proxy is used to load balance these S3 Gateway
> Servers.
>
> In this Jira, all are proxy configurations are hardcoded, we can make
> improvements to scale and automatically configure with environment variables
> as a future improvement. This is just a starter example.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1972) Provide example ha proxy with multiple s3 servers back end.
[ https://issues.apache.org/jira/browse/HDDS-1972?focusedWorklogId=297411=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297411 ] ASF GitHub Bot logged work on HDDS-1972: Author: ASF GitHub Bot Created on: 19/Aug/19 21:00 Start Date: 19/Aug/19 21:00 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #1304: HDDS-1972. Provide example ha proxy with multiple s3 servers back end. URL: https://github.com/apache/hadoop/pull/1304 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297411) Time Spent: 4h 40m (was: 4.5h) > Provide example ha proxy with multiple s3 servers back end. > --- > > Key: HDDS-1972 > URL: https://issues.apache.org/jira/browse/HDDS-1972 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 4h 40m > Remaining Estimate: 0h > > In this Jira, we shall provide docker-compose files where we start 3 s3 > gateway servers, and ha-proxy is used to load balance these S3 Gateway > Servers. > > In this Jira, all are proxy configurations are hardcoded, we can make > improvements to scale and automatically configure with environment variables > as a future improvement. This is just a starter example. > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1975) Implement default acls for bucket/volume/key for OM HA code
[ https://issues.apache.org/jira/browse/HDDS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1975: - Target Version/s: 0.5.0 > Implement default acls for bucket/volume/key for OM HA code > --- > > Key: HDDS-1975 > URL: https://issues.apache.org/jira/browse/HDDS-1975 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to implement default ACLs for Ozone volume/bucket/key. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-14665) HttpFS: LISTSTATUS response is missing HDFS-specific fields
[
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang reopened HDFS-14665:
Reopen for branch-3.1
> HttpFS: LISTSTATUS response is missing HDFS-specific fields
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Fix For: 3.3.0, 3.2.1
>
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
> }
> ]
> }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
> }
> ]
> }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (should only be none 0 for directories)
> "fileId"
> "storagePolicy"
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same
> underlying calls to compose the response.
> Root cause:
> [toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
> didn't serialize the HDFS-specific keys from FileStatus.
> Also may file another Jira to align the order of the keys in the responses.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12510) RBF: Add security to UI
[ https://issues.apache.org/jira/browse/HDFS-12510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910722#comment-16910722 ] Íñigo Goiri commented on HDFS-12510: Let's mark it done for now. > RBF: Add security to UI > --- > > Key: HDFS-12510 > URL: https://issues.apache.org/jira/browse/HDFS-12510 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Labels: RBF > > HDFS-12273 implemented the UI for Router Based Federation without security. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1972) Provide example ha proxy with multiple s3 servers back end.
[ https://issues.apache.org/jira/browse/HDDS-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1972: - Fix Version/s: 0.5.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Provide example ha proxy with multiple s3 servers back end. > --- > > Key: HDDS-1972 > URL: https://issues.apache.org/jira/browse/HDDS-1972 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > In this Jira, we shall provide docker-compose files where we start 3 s3 > gateway servers, and ha-proxy is used to load balance these S3 Gateway > Servers. > > In this Jira, all are proxy configurations are hardcoded, we can make > improvements to scale and automatically configure with environment variables > as a future improvement. This is just a starter example. > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1972) Provide example ha proxy with multiple s3 servers back end.
[ https://issues.apache.org/jira/browse/HDDS-1972?focusedWorklogId=297412=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297412 ] ASF GitHub Bot logged work on HDDS-1972: Author: ASF GitHub Bot Created on: 19/Aug/19 21:01 Start Date: 19/Aug/19 21:01 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #1304: HDDS-1972. Provide example ha proxy with multiple s3 servers back end. URL: https://github.com/apache/hadoop/pull/1304#issuecomment-522752048 > +1 > > @adoroszlai are you okay to commit this? Got a +1 from @adoroszlai already, so committed this change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297412) Time Spent: 4h 50m (was: 4h 40m) > Provide example ha proxy with multiple s3 servers back end. > --- > > Key: HDDS-1972 > URL: https://issues.apache.org/jira/browse/HDDS-1972 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > In this Jira, we shall provide docker-compose files where we start 3 s3 > gateway servers, and ha-proxy is used to load balance these S3 Gateway > Servers. > > In this Jira, all are proxy configurations are hardcoded, we can make > improvements to scale and automatically configure with environment variables > as a future improvement. This is just a starter example. > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14746) Trivial test code update after HDFS-14687
[
https://issues.apache.org/jira/browse/HDFS-14746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910639#comment-16910639
]
Hudson commented on HDFS-14746:
---
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17148 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17148/])
HDFS-14746. Trivial test code update after HDFS-14687. Contributed by (weichiu:
rev abae6ff2a2760500b7e7d4414a43069ed4a45930)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingDataNodeMessages.java
> Trivial test code update after HDFS-14687
> -
>
> Key: HDFS-14746
> URL: https://issues.apache.org/jira/browse/HDFS-14746
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ec
>Reporter: Wei-Chiu Chuang
>Assignee: kevin su
>Priority: Trivial
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14746.001.patch
>
>
> Instead of getting erasure coding policy instance by id, it should use a
> constant value.
> Change
> {code}
> ErasureCodingPolicy ecPolicy = SystemErasureCodingPolicies.getPolicies()
> .get(3);
> {code}
> to
> {code}
> ErasureCodingPolicy ecPolicy =
> SystemErasureCodingPolicies.getByID(XOR_2_1_POLICY_ID);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14687) Standby Namenode never come out of safemode when EC files are being written.
[
https://issues.apache.org/jira/browse/HDFS-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910640#comment-16910640
]
Hudson commented on HDFS-14687:
---
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17148 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17148/])
HDFS-14746. Trivial test code update after HDFS-14687. Contributed by (weichiu:
rev abae6ff2a2760500b7e7d4414a43069ed4a45930)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingDataNodeMessages.java
> Standby Namenode never come out of safemode when EC files are being written.
>
>
> Key: HDFS-14687
> URL: https://issues.apache.org/jira/browse/HDFS-14687
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, namenode
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Critical
> Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14687.001.patch, HDFS-14687.002.patch,
> HDFS-14687.003.patch, HDFS-14687.004.patch
>
>
> When huge number of EC files are being written and SBN is restarted then it
> will never come out of safe mode and required blocks count getting increase.
> {noformat}
> The reported blocks 16658401 needs additional 1702 blocks to reach the
> threshold 0.9 of total blocks 16660120.
> The reported blocks 16658659 needs additional 2935 blocks to reach the
> threshold 0.9 of total blocks 16661611.
> The reported blocks 16659947 needs additional 3868 blocks to reach the
> threshold 0.9 of total blocks 16663832.
> The reported blocks 1335 needs additional 5116 blocks to reach the
> threshold 0.9 of total blocks 16671468.
> The reported blocks 16669311 needs additional 6384 blocks to reach the
> threshold 0.9 of total blocks 16675712.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1972) Provide example ha proxy with multiple s3 servers back end.
[ https://issues.apache.org/jira/browse/HDDS-1972?focusedWorklogId=297371=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297371 ] ASF GitHub Bot logged work on HDDS-1972: Author: ASF GitHub Bot Created on: 19/Aug/19 19:31 Start Date: 19/Aug/19 19:31 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1304: HDDS-1972. Provide example ha proxy with multiple s3 servers back end. URL: https://github.com/apache/hadoop/pull/1304#issuecomment-522721525 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 46 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | 0 | yamllint | 0 | yamllint was not available. | | 0 | shelldocs | 0 | Shelldocs was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 601 | trunk passed | | +1 | compile | 375 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 723 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 158 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 566 | the patch passed | | +1 | compile | 395 | the patch passed | | +1 | javac | 395 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | shellcheck | 0 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 652 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 170 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 316 | hadoop-hdds in the patch passed. | | -1 | unit | 2696 | hadoop-ozone in the patch failed. | | +1 | asflicense | 49 | The patch does not generate ASF License warnings. | | | | 6969 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.dn.scrubber.TestDataScrubber | | | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | | hadoop.ozone.client.rpc.TestFailureHandlingByClient | | | hadoop.ozone.client.rpc.TestCommitWatcher | | | hadoop.ozone.client.rpc.TestReadRetries | | | hadoop.ozone.om.TestOzoneManagerHA | | | hadoop.ozone.client.rpc.TestHybridPipelineOnDatanode | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1304 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient yamllint shellcheck shelldocs | | uname | Linux a05c19df5e06 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / c765584 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/7/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/7/testReport/ | | Max. process+thread count | 3723 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/dist U: hadoop-ozone/dist | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/7/console | | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297371) Time Spent: 4h (was: 3h 50m) > Provide example ha proxy with multiple s3 servers back end. > --- > > Key: HDDS-1972 > URL: https://issues.apache.org/jira/browse/HDDS-1972 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available >
[jira] [Commented] (HDFS-14090) RBF: Improved isolation for downstream name nodes.
[
https://issues.apache.org/jira/browse/HDFS-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910708#comment-16910708
]
CR Hota commented on HDFS-14090:
[~hexiaoqiao] Thanks for the review. The points you raised are very valid.
In the design doc also I have mentioned that at some point we need to introduce
and look into preemption/dynamic allocation. Yes, but for Phase 1 the current
patch will help installations move forward with the concept of isolation.
Dynamic/Preemption will obviously be a separate implementation of
{{FairnessPolicyController}}. I will open a ticket to track this next phase.
This would also need a through design analysis and review.
Lets wait for [~elgoiri] [~brahmareddy] [~aajisaka] [~xkrogen] to review the
010 patch.
> RBF: Improved isolation for downstream name nodes.
> --
>
> Key: HDFS-14090
> URL: https://issues.apache.org/jira/browse/HDFS-14090
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14090-HDFS-13891.001.patch,
> HDFS-14090-HDFS-13891.002.patch, HDFS-14090-HDFS-13891.003.patch,
> HDFS-14090-HDFS-13891.004.patch, HDFS-14090-HDFS-13891.005.patch,
> HDFS-14090.006.patch, HDFS-14090.007.patch, HDFS-14090.008.patch,
> HDFS-14090.009.patch, HDFS-14090.010.patch, RBF_ Isolation design.pdf
>
>
> Router is a gateway to underlying name nodes. Gateway architectures, should
> help minimize impact of clients connecting to healthy clusters vs unhealthy
> clusters.
> For example - If there are 2 name nodes downstream, and one of them is
> heavily loaded with calls spiking rpc queue times, due to back pressure the
> same with start reflecting on the router. As a result of this, clients
> connecting to healthy/faster name nodes will also slow down as same rpc queue
> is maintained for all calls at the router layer. Essentially the same IPC
> thread pool is used by router to connect to all name nodes.
> Currently router uses one single rpc queue for all calls. Lets discuss how we
> can change the architecture and add some throttling logic for
> unhealthy/slow/overloaded name nodes.
> One way could be to read from current call queue, immediately identify
> downstream name node and maintain a separate queue for each underlying name
> node. Another simpler way is to maintain some sort of rate limiter configured
> for each name node and let routers drop/reject/send error requests after
> certain threshold.
> This won’t be a simple change as router’s ‘Server’ layer would need redesign
> and implementation. Currently this layer is the same as name node.
> Opening this ticket to discuss, design and implement this feature.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14750) RBF: Improved isolation for downstream name nodes. {Dynamic}
CR Hota created HDFS-14750:
--
Summary: RBF: Improved isolation for downstream name nodes.
{Dynamic}
Key: HDFS-14750
URL: https://issues.apache.org/jira/browse/HDFS-14750
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: CR Hota
Assignee: CR Hota
This Jira tracks the work around dynamic allocation of resources in routers for
downstream hdfs clusters.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13709) Report bad block to NN when transfer block encounter EIO exception
[ https://issues.apache.org/jira/browse/HDFS-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910744#comment-16910744 ] Hudson commented on HDFS-13709: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17149 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17149/]) HDFS-13709. Report bad block to NN when transfer block encounter EIO (weichiu: rev 360a96f342f3c8cb8246f011abb9bcb0b6ef3eaa) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskFileCorruptException.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReplication.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java > Report bad block to NN when transfer block encounter EIO exception > -- > > Key: HDFS-13709 > URL: https://issues.apache.org/jira/browse/HDFS-13709 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Chen Zhang >Assignee: Chen Zhang >Priority: Major > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: HDFS-13709.002.patch, HDFS-13709.003.patch, > HDFS-13709.004.patch, HDFS-13709.005.patch, HDFS-13709.patch > > > In our online cluster, the BlockPoolSliceScanner is turned off, and sometimes > disk bad track may cause data loss. > For example, there are 3 replicas on 3 machines A/B/C, if a bad track occurs > on A's replica data, and someday B and C crushed at the same time, NN will > try to replicate data from A but failed, this block is corrupt now but no one > knows, because NN think there is at least 1 healthy replica and it keep > trying to replicate it. > When reading a replica which have data on bad track, OS will return an EIO > error, if DN reports the bad block as soon as it got an EIO, we can find > this case ASAP and try to avoid data loss -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14090) RBF: Improved isolation for downstream name nodes. {Static}
[
https://issues.apache.org/jira/browse/HDFS-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
CR Hota updated HDFS-14090:
---
Summary: RBF: Improved isolation for downstream name nodes. {Static} (was:
RBF: Improved isolation for downstream name nodes.)
> RBF: Improved isolation for downstream name nodes. {Static}
> ---
>
> Key: HDFS-14090
> URL: https://issues.apache.org/jira/browse/HDFS-14090
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14090-HDFS-13891.001.patch,
> HDFS-14090-HDFS-13891.002.patch, HDFS-14090-HDFS-13891.003.patch,
> HDFS-14090-HDFS-13891.004.patch, HDFS-14090-HDFS-13891.005.patch,
> HDFS-14090.006.patch, HDFS-14090.007.patch, HDFS-14090.008.patch,
> HDFS-14090.009.patch, HDFS-14090.010.patch, RBF_ Isolation design.pdf
>
>
> Router is a gateway to underlying name nodes. Gateway architectures, should
> help minimize impact of clients connecting to healthy clusters vs unhealthy
> clusters.
> For example - If there are 2 name nodes downstream, and one of them is
> heavily loaded with calls spiking rpc queue times, due to back pressure the
> same with start reflecting on the router. As a result of this, clients
> connecting to healthy/faster name nodes will also slow down as same rpc queue
> is maintained for all calls at the router layer. Essentially the same IPC
> thread pool is used by router to connect to all name nodes.
> Currently router uses one single rpc queue for all calls. Lets discuss how we
> can change the architecture and add some throttling logic for
> unhealthy/slow/overloaded name nodes.
> One way could be to read from current call queue, immediately identify
> downstream name node and maintain a separate queue for each underlying name
> node. Another simpler way is to maintain some sort of rate limiter configured
> for each name node and let routers drop/reject/send error requests after
> certain threshold.
> This won’t be a simple change as router’s ‘Server’ layer would need redesign
> and implementation. Currently this layer is the same as name node.
> Opening this ticket to discuss, design and implement this feature.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1975) Implement default acls for bucket/volume/key for OM HA code
[ https://issues.apache.org/jira/browse/HDDS-1975?focusedWorklogId=297403=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297403 ] ASF GitHub Bot logged work on HDDS-1975: Author: ASF GitHub Bot Created on: 19/Aug/19 20:37 Start Date: 19/Aug/19 20:37 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #1315: HDDS-1975. Implement default acls for bucket/volume/key for OM HA code. URL: https://github.com/apache/hadoop/pull/1315 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297403) Remaining Estimate: 0h Time Spent: 10m > Implement default acls for bucket/volume/key for OM HA code > --- > > Key: HDDS-1975 > URL: https://issues.apache.org/jira/browse/HDDS-1975 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to implement default ACLs for Ozone volume/bucket/key. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1975) Implement default acls for bucket/volume/key for OM HA code
[ https://issues.apache.org/jira/browse/HDDS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1975: - Labels: pull-request-available (was: ) > Implement default acls for bucket/volume/key for OM HA code > --- > > Key: HDDS-1975 > URL: https://issues.apache.org/jira/browse/HDDS-1975 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > This Jira is to implement default ACLs for Ozone volume/bucket/key. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1975) Implement default acls for bucket/volume/key for OM HA code
[ https://issues.apache.org/jira/browse/HDDS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1975: - Status: Patch Available (was: Open) > Implement default acls for bucket/volume/key for OM HA code > --- > > Key: HDDS-1975 > URL: https://issues.apache.org/jira/browse/HDDS-1975 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to implement default ACLs for Ozone volume/bucket/key. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1986) Fix listkeys API
Bharat Viswanadham created HDDS-1986: Summary: Fix listkeys API Key: HDDS-1986 URL: https://issues.apache.org/jira/browse/HDDS-1986 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Bharat Viswanadham This Jira is to fix listBucket API in HA code path. In HA, we have an in-memory cache, where we put the result to in-memory cache and return the response, later it will be picked by double buffer thread and it will flush to disk. So, now when do listBuckets, it should use both in-memory cache and rocksdb bucket table to list buckets in a volume. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1985) Fix listVolumes API
[ https://issues.apache.org/jira/browse/HDDS-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1985: - Description: This Jira is to fix lisVolumes API in HA code path. In HA, we have an in-memory cache, where we put the result to in-memory cache and return the response, later it will be picked by double buffer thread and it will flush to disk. So, now when do listVolumes, it should use both in-memory cache and rocksdb volume table to list volumes for a user. was: This Jira is to fix listBucket API in HA code path. In HA, we have an in-memory cache, where we put the result to in-memory cache and return the response, later it will be picked by double buffer thread and it will flush to disk. So, now when do listBuckets, it should use both in-memory cache and rocksdb bucket table to list buckets in a volume. > Fix listVolumes API > --- > > Key: HDDS-1985 > URL: https://issues.apache.org/jira/browse/HDDS-1985 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Priority: Major > > This Jira is to fix lisVolumes API in HA code path. > In HA, we have an in-memory cache, where we put the result to in-memory cache > and return the response, later it will be picked by double buffer thread and > it will flush to disk. So, now when do listVolumes, it should use both > in-memory cache and rocksdb volume table to list volumes for a user. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12241) HttpFS to support overloaded FileSystem#rename API
[
https://issues.apache.org/jira/browse/HDFS-12241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun reassigned HDFS-12241:
--
Assignee: Lisheng Sun (was: Wei-Chiu Chuang)
> HttpFS to support overloaded FileSystem#rename API
> --
>
> Key: HDFS-12241
> URL: https://issues.apache.org/jira/browse/HDFS-12241
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: httpfs
>Reporter: Wei-Chiu Chuang
>Assignee: Lisheng Sun
>Priority: Major
>
> Httpfs is essentially the parity of webhdfs. But it does not implement
> {{FileSystem#rename(final Path src, final Path dst, final Rename...
> options)}}, which mean it does not support trash.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13709) Report bad block to NN when transfer block encounter EIO exception
[ https://issues.apache.org/jira/browse/HDFS-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910630#comment-16910630 ] Wei-Chiu Chuang commented on HDFS-13709: +1 > Report bad block to NN when transfer block encounter EIO exception > -- > > Key: HDFS-13709 > URL: https://issues.apache.org/jira/browse/HDFS-13709 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Chen Zhang >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-13709.002.patch, HDFS-13709.003.patch, > HDFS-13709.004.patch, HDFS-13709.005.patch, HDFS-13709.patch > > > In our online cluster, the BlockPoolSliceScanner is turned off, and sometimes > disk bad track may cause data loss. > For example, there are 3 replicas on 3 machines A/B/C, if a bad track occurs > on A's replica data, and someday B and C crushed at the same time, NN will > try to replicate data from A but failed, this block is corrupt now but no one > knows, because NN think there is at least 1 healthy replica and it keep > trying to replicate it. > When reading a replica which have data on bad track, OS will return an EIO > error, if DN reports the bad block as soon as it got an EIO, we can find > this case ASAP and try to avoid data loss -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14567) If kms-acls is failed to load, and it will never be reload
[
https://issues.apache.org/jira/browse/HDFS-14567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hemanthboyina updated HDFS-14567:
-
Attachment: HDFS-14567.001.patch
> If kms-acls is failed to load, and it will never be reload
> ---
>
> Key: HDFS-14567
> URL: https://issues.apache.org/jira/browse/HDFS-14567
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS-14567.001.patch, HDFS-14567.patch
>
>
> Scenario : through one automation tool , we are generating kms-acls , though
> the generation of kms-acls is not completed , the system will detect a
> modification of kms-alcs and it will try to load
> Before getting the configuration we are modifiying last reload time , code
> shown below
> {code:java}
> private Configuration loadACLsFromFile() {
> LOG.debug("Loading ACLs file");
> lastReload = System.currentTimeMillis();
> Configuration conf = KMSConfiguration.getACLsConf();
> // triggering the resource loading.
> conf.get(Type.CREATE.getAclConfigKey());
> return conf;
> }{code}
> if the kms-acls file written within next 100ms , the changes will not be
> loaded as this condition "newer = f.lastModified() - time > 100" never meets
> because we have modified last reload time before getting the configuration
> {code:java}
> public static boolean isACLsFileNewer(long time) {
> boolean newer = false;
> String confDir = System.getProperty(KMS_CONFIG_DIR);
> if (confDir != null) {
> Path confPath = new Path(confDir);
> if (!confPath.isUriPathAbsolute()) {
> throw new RuntimeException("System property '" + KMS_CONFIG_DIR +
> "' must be an absolute path: " + confDir);
> }
> File f = new File(confDir, KMS_ACLS_XML);
> LOG.trace("Checking file {}, modification time is {}, last reload time is"
> + " {}", f.getPath(), f.lastModified(), time);
> // at least 100ms newer than time, we do this to ensure the file
> // has been properly closed/flushed
> newer = f.lastModified() - time > 100;
> }
> return newer;
> } {code}
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910617#comment-16910617 ] Hudson commented on HDDS-1105: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17147 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17147/]) HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates (hanishakoneru: rev d69a1a0aa49614c084fa4b9546ace65aebe4) * (edit) hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/spi/impl/TestOzoneManagerServiceProviderImpl.java * (edit) hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestUtilizationService.java * (edit) hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/DummyReconDBTask.java * (edit) hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/AbstractOMMetadataManagerTest.java * (edit) hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestContainerKeyMapperTask.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OMUpdateEventBatch.java * (edit) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ContainerKeyMapperTask.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ReconTaskController.java * (edit) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java * (edit) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java * (edit) hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/TestReconUtils.java * (edit) hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestContainerKeyService.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconUtils.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBUpdatesWrapper.java * (edit) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconControllerModule.java * (add) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconTaskBindingModule.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconContainerDBProvider.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ContainerDBServiceProviderImpl.java * (edit) hadoop-ozone/ozone-recon/pom.xml * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServer.java * (edit) hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestReconTaskControllerImpl.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/FileSizeCountTask.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OMDBUpdateEvent.java * (edit) hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestFileSizeCountTask.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/OzoneManagerServiceProviderImpl.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServerConfigKeys.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBBatchOperation.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/spi/OzoneManagerServiceProvider.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OMDBUpdatesHandler.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ReconDBUpdateTask.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ReconTaskControllerImpl.java > Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone > Manager. > > > Key: HDDS-1105 > URL: https://issues.apache.org/jira/browse/HDDS-1105 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > *Some context* > The FSCK server will periodically invoke this OM API passing in the most > recent sequence number of its own RocksDB instance. The OM will use the > RockDB getUpdateSince() API to answer this query. Since the getUpdateSince > API only works against the RocksDB WAL, we have to configure OM RocksDB WAL >
[jira] [Updated] (HDFS-14746) Trivial test code update after HDFS-14687
[
https://issues.apache.org/jira/browse/HDFS-14746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14746:
---
Fix Version/s: 3.1.3
3.2.1
3.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks all!
> Trivial test code update after HDFS-14687
> -
>
> Key: HDFS-14746
> URL: https://issues.apache.org/jira/browse/HDFS-14746
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ec
>Reporter: Wei-Chiu Chuang
>Assignee: kevin su
>Priority: Trivial
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14746.001.patch
>
>
> Instead of getting erasure coding policy instance by id, it should use a
> constant value.
> Change
> {code}
> ErasureCodingPolicy ecPolicy = SystemErasureCodingPolicies.getPolicies()
> .get(3);
> {code}
> to
> {code}
> ErasureCodingPolicy ecPolicy =
> SystemErasureCodingPolicies.getByID(XOR_2_1_POLICY_ID);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14174) Enhance Audit for chown ( internally setOwner)
[ https://issues.apache.org/jira/browse/HDFS-14174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910486#comment-16910486 ] hemanthboyina commented on HDFS-14174: -- when a user operation done , for a major change (in users perspective) like setting owner It needs to be audited that , what was before and what was now (owner) Just to track out, it was required . > Enhance Audit for chown ( internally setOwner) > - > > Key: HDFS-14174 > URL: https://issues.apache.org/jira/browse/HDFS-14174 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.6.0 >Reporter: Sailesh Patel >Assignee: hemanthboyina >Priority: Minor > > When a hdfs dfs -chown command is executed, the audit log does not capture > the existing owner and the new owner. > Need to capture the old and new owner to allow auditing to be effective > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14748) Make DataNodePeerMetrics#minOutlierDetectionSamples configurable
[
https://issues.apache.org/jira/browse/HDFS-14748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910607#comment-16910607
]
Hadoop QA commented on HDFS-14748:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 46s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 606 unchanged - 0 fixed = 607 total (was 606) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
9s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 10s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}162m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Write to static field
org.apache.hadoop.hdfs.server.datanode.metrics.DataNodePeerMetrics.minOutlierDetectionSamples
from instance method new
org.apache.hadoop.hdfs.server.datanode.metrics.DataNodePeerMetrics(String,
Configuration) At DataNodePeerMetrics.java:from instance method new
org.apache.hadoop.hdfs.server.datanode.metrics.DataNodePeerMetrics(String,
Configuration) At DataNodePeerMetrics.java:[line 70] |
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.TestMultipleNNPortQOP |
| | hadoop.hdfs.server.namenode.TestFsck |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HDFS-14748 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977980/HDFS-14748.001.patch |
| Optional Tests | dupname asflicense compile javac
[jira] [Created] (HDDS-1983) Enable S3 test suite on S3proxy cluster
Bharat Viswanadham created HDDS-1983: Summary: Enable S3 test suite on S3proxy cluster Key: HDDS-1983 URL: https://issues.apache.org/jira/browse/HDDS-1983 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham The S3 test suite is disabled for s3 proxy cluster. (This is because it is consistenly failing in Jenkins run, but I was able to run it locally. This Jira is to figure out any changes required to enable s3 test suite on s3 proxy cluster which is brought in HDDS-1972. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14501) BenchmarkThroughput.writeFile hangs with misconfigured BUFFER_SIZE
[
https://issues.apache.org/jira/browse/HDFS-14501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hemanthboyina reassigned HDFS-14501:
Assignee: hemanthboyina
> BenchmarkThroughput.writeFile hangs with misconfigured BUFFER_SIZE
> --
>
> Key: HDFS-14501
> URL: https://issues.apache.org/jira/browse/HDFS-14501
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: John Doe
>Assignee: hemanthboyina
>Priority: Major
>
> When the configuration file is corrupted, reading BUFFER_SIZE from corrupted
> conf can return 0.
> The "for" loop in BenchmarkThroughput.writeLocalFile function hangs
> endlessly.
> Here is the code snippet.
> {code:java}
> BUFFER_SIZE = conf.getInt("dfsthroughput.buffer.size", 4 * 1024);
> private Path writeFile(FileSystem fs,
> String name,
> Configuration conf,
> long total
> ) throws IOException {
> Path f = dir.getLocalPathForWrite(name, total, conf);
> System.out.print("Writing " + name);
> resetMeasurements();
> OutputStream out = fs.create(f);
> byte[] data = new byte[BUFFER_SIZE];
> for(long size = 0; size < total; size += BUFFER_SIZE) { //Bug!
> out.write(data);
> }
> out.close();
> printMeasurements();
> return f;
> }
> {code}
> This configuration error also affects HDFS-13513, HDFS-13514,
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14714) RBF: implement getReplicatedBlockStats interface
[
https://issues.apache.org/jira/browse/HDFS-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Zhang updated HDFS-14714:
--
Status: Patch Available (was: Open)
> RBF: implement getReplicatedBlockStats interface
>
>
> Key: HDFS-14714
> URL: https://issues.apache.org/jira/browse/HDFS-14714
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
> Attachments: HDFS-14714.001.patch
>
>
> It's not implemented now, we sometime need this interface for cluster monitor
> {code:java}
> // current implementation
> public ReplicatedBlockStats getReplicatedBlockStats() throws IOException {
> rpcServer.checkOperation(NameNode.OperationCategory.READ, false);
> return null;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14714) RBF: implement getReplicatedBlockStats interface
[
https://issues.apache.org/jira/browse/HDFS-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910546#comment-16910546
]
Chen Zhang commented on HDFS-14714:
---
Thanks [~ayushtkn] for your suggestion, uploaded the patch v1
> RBF: implement getReplicatedBlockStats interface
>
>
> Key: HDFS-14714
> URL: https://issues.apache.org/jira/browse/HDFS-14714
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
> Attachments: HDFS-14714.001.patch
>
>
> It's not implemented now, we sometime need this interface for cluster monitor
> {code:java}
> // current implementation
> public ReplicatedBlockStats getReplicatedBlockStats() throws IOException {
> rpcServer.checkOperation(NameNode.OperationCategory.READ, false);
> return null;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14714) RBF: implement getReplicatedBlockStats interface
[
https://issues.apache.org/jira/browse/HDFS-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Zhang updated HDFS-14714:
--
Attachment: HDFS-14714.001.patch
> RBF: implement getReplicatedBlockStats interface
>
>
> Key: HDFS-14714
> URL: https://issues.apache.org/jira/browse/HDFS-14714
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
> Attachments: HDFS-14714.001.patch
>
>
> It's not implemented now, we sometime need this interface for cluster monitor
> {code:java}
> // current implementation
> public ReplicatedBlockStats getReplicatedBlockStats() throws IOException {
> rpcServer.checkOperation(NameNode.OperationCategory.READ, false);
> return null;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1982) Extend SCMNodeManager to support decommission and maintenance states
[ https://issues.apache.org/jira/browse/HDDS-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910562#comment-16910562 ] Anu Engineer commented on HDDS-1982: +1. This allows us to add new states more logically later. For example, when we add support for administrator approval of all certificates issued, a data node would be joined to the cluster but not operational since the administrator has not approved the datanode certificate yet. In that case, we could name that state as QUARANTINE, in other words, it will be easier for us to extend this matrix with this approach. > Extend SCMNodeManager to support decommission and maintenance states > > > Key: HDDS-1982 > URL: https://issues.apache.org/jira/browse/HDDS-1982 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > > Currently, within SCM a node can have the following states: > HEALTHY > STALE > DEAD > DECOMMISSIONING > DECOMMISSIONED > The last 2 are not currently used. > In order to support decommissioning and maintenance mode, we need to extend > the set of states a node can have to include decommission and maintenance > states. > It is also important to note that a node decommissioning or entering > maintenance can also be HEALTHY, STALE or go DEAD. > Therefore in this Jira I propose we should model a node state with two > different sets of values. The first, is effectively the liveliness of the > node, with the following states. This is largely what is in place now: > HEALTHY > STALE > DEAD > The second is the node operational state: > IN_SERVICE > DECOMMISSIONING > DECOMMISSIONED > ENTERING_MAINTENANCE > IN_MAINTENANCE > That means the overall total number of states for a node is the cross-product > of the two above lists, however it probably makes sense to keep the two > states seperate internally. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14729) Upgrade Bootstrap and jQuery versions used in HDFS UIs
[ https://issues.apache.org/jira/browse/HDFS-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vivek Ratnavel Subramanian updated HDFS-14729: -- Attachment: HDFS-14729.v1.patch > Upgrade Bootstrap and jQuery versions used in HDFS UIs > -- > > Key: HDFS-14729 > URL: https://issues.apache.org/jira/browse/HDFS-14729 > Project: Hadoop HDFS > Issue Type: Task > Components: ui >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Attachments: HDFS-14729.v1.patch > > > The current versions of bootstrap and jquery have multiple medium severity > CVEs reported till date and needs to be updated to the latest versions with > no reported CVEs. > > I suggest updating the following libraries: > ||Library||From version||To version|| > |Bootstrap|3.3.7|3.4.1| > |jQuery|3.3.1|3.4.1| -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14729) Upgrade Bootstrap and jQuery versions used in HDFS UIs
[ https://issues.apache.org/jira/browse/HDFS-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vivek Ratnavel Subramanian updated HDFS-14729: -- Status: Patch Available (was: In Progress) > Upgrade Bootstrap and jQuery versions used in HDFS UIs > -- > > Key: HDFS-14729 > URL: https://issues.apache.org/jira/browse/HDFS-14729 > Project: Hadoop HDFS > Issue Type: Task > Components: ui >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Attachments: HDFS-14729.v1.patch > > > The current versions of bootstrap and jquery have multiple medium severity > CVEs reported till date and needs to be updated to the latest versions with > no reported CVEs. > > I suggest updating the following libraries: > ||Library||From version||To version|| > |Bootstrap|3.3.7|3.4.1| > |jQuery|3.3.1|3.4.1| -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14567) If kms-acls is failed to load, and it will never be reload
[
https://issues.apache.org/jira/browse/HDFS-14567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910591#comment-16910591
]
Hadoop QA commented on HDFS-14567:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
53s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
20s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
1s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}101m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HDFS-14567 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977981/HDFS-14567.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9eeeb4acf75c 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c765584 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/27564/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-kms U:
hadoop-common-project/hadoop-kms |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/27564/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> If kms-acls is failed to load, and it will never be reload
> ---
>
> Key: HDFS-14567
> URL:
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=297311=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297311 ] ASF GitHub Bot logged work on HDDS-1105: Author: ASF GitHub Bot Created on: 19/Aug/19 17:22 Start Date: 19/Aug/19 17:22 Worklog Time Spent: 10m Work Description: hanishakoneru commented on issue #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259#issuecomment-522674063 Thank you @avijayanhwx. +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297311) Time Spent: 3h 20m (was: 3h 10m) > Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone > Manager. > > > Key: HDDS-1105 > URL: https://issues.apache.org/jira/browse/HDDS-1105 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > *Some context* > The FSCK server will periodically invoke this OM API passing in the most > recent sequence number of its own RocksDB instance. The OM will use the > RockDB getUpdateSince() API to answer this query. Since the getUpdateSince > API only works against the RocksDB WAL, we have to configure OM RocksDB WAL > (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log) with sufficient > max size to make this API useful. If the OM cannot get all transactions since > the given sequence number (due to WAL flushing), it can error out. In that > case the FSCK server can fall back to getting the entire checkpoint snapshot > implemented in HDDS-1085. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1972) Provide example ha proxy with multiple s3 servers back end.
[ https://issues.apache.org/jira/browse/HDDS-1972?focusedWorklogId=297322=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297322 ] ASF GitHub Bot logged work on HDDS-1972: Author: ASF GitHub Bot Created on: 19/Aug/19 17:37 Start Date: 19/Aug/19 17:37 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #1304: HDDS-1972. Provide example ha proxy with multiple s3 servers back end. URL: https://github.com/apache/hadoop/pull/1304#issuecomment-522679586 I see tests are passing locally on my laptop. I was able to test ha-proxy with aws cli. Not able to figure out why on jenkins it is showing not able to connect to http://s3g:9878. For now, I have disabled the test. But this will give an example for users to use ha-proxy setup with S3Gateway Server. (I will open a new Jira to enable s3 test suite for s3 proxy) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297322) Time Spent: 3h 40m (was: 3.5h) > Provide example ha proxy with multiple s3 servers back end. > --- > > Key: HDDS-1972 > URL: https://issues.apache.org/jira/browse/HDDS-1972 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 3h 40m > Remaining Estimate: 0h > > In this Jira, we shall provide docker-compose files where we start 3 s3 > gateway servers, and ha-proxy is used to load balance these S3 Gateway > Servers. > > In this Jira, all are proxy configurations are hardcoded, we can make > improvements to scale and automatically configure with environment variables > as a future improvement. This is just a starter example. > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-1105: - Resolution: Fixed Status: Resolved (was: Patch Available) > Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone > Manager. > > > Key: HDDS-1105 > URL: https://issues.apache.org/jira/browse/HDDS-1105 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > *Some context* > The FSCK server will periodically invoke this OM API passing in the most > recent sequence number of its own RocksDB instance. The OM will use the > RockDB getUpdateSince() API to answer this query. Since the getUpdateSince > API only works against the RocksDB WAL, we have to configure OM RocksDB WAL > (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log) with sufficient > max size to make this API useful. If the OM cannot get all transactions since > the given sequence number (due to WAL flushing), it can error out. In that > case the FSCK server can fall back to getting the entire checkpoint snapshot > implemented in HDDS-1085. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14746) Trivial test code update after HDFS-14687
[
https://issues.apache.org/jira/browse/HDFS-14746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910492#comment-16910492
]
Wei-Chiu Chuang commented on HDFS-14746:
Not sure why Yetus didn't report result, but this run in
https://builds.apache.org/job/PreCommit-HDFS-Build/27558/
Test failure looks unrelated. +1 I'll commit soon.
> Trivial test code update after HDFS-14687
> -
>
> Key: HDFS-14746
> URL: https://issues.apache.org/jira/browse/HDFS-14746
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ec
>Reporter: Wei-Chiu Chuang
>Assignee: kevin su
>Priority: Trivial
> Attachments: HDFS-14746.001.patch
>
>
> Instead of getting erasure coding policy instance by id, it should use a
> constant value.
> Change
> {code}
> ErasureCodingPolicy ecPolicy = SystemErasureCodingPolicies.getPolicies()
> .get(3);
> {code}
> to
> {code}
> ErasureCodingPolicy ecPolicy =
> SystemErasureCodingPolicies.getByID(XOR_2_1_POLICY_ID);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1972) Provide example ha proxy with multiple s3 servers back end.
[ https://issues.apache.org/jira/browse/HDDS-1972?focusedWorklogId=297329=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297329 ] ASF GitHub Bot logged work on HDDS-1972: Author: ASF GitHub Bot Created on: 19/Aug/19 17:45 Start Date: 19/Aug/19 17:45 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #1304: HDDS-1972. Provide example ha proxy with multiple s3 servers back end. URL: https://github.com/apache/hadoop/pull/1304#issuecomment-522679586 I see tests are passing locally on my laptop. I was able to test ha-proxy with aws cli. Not able to figure out why on jenkins it is showing not able to connect to http://s3g:9878. For now, I have disabled the test. But this will give an example for users to use ha-proxy setup with S3Gateway Server. (I will open a new Jira(https://issues.apache.org/jira/browse/HDDS-1983) to enable s3 test suite for s3 proxy) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297329) Time Spent: 3h 50m (was: 3h 40m) > Provide example ha proxy with multiple s3 servers back end. > --- > > Key: HDDS-1972 > URL: https://issues.apache.org/jira/browse/HDDS-1972 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > In this Jira, we shall provide docker-compose files where we start 3 s3 > gateway servers, and ha-proxy is used to load balance these S3 Gateway > Servers. > > In this Jira, all are proxy configurations are hardcoded, we can make > improvements to scale and automatically configure with environment variables > as a future improvement. This is just a starter example. > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14609) RBF: Security should use common AuthenticationFilter
[ https://issues.apache.org/jira/browse/HDFS-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910618#comment-16910618 ] CR Hota commented on HDFS-14609: [~zhangchen] Thanks for the patch. Few points # Lets not add hadoop-hdfs and rbf changes together. For any changes needed to hdfs that rbf depends on, can be first done in hdfs through a separate jira. Feel free to create one. # It's not clear why hadoop-hdfs changes are needed in this context. # Let's use the configs from DFSConfigs instead of defining them in the test class again for ex : private static final String HTTP_KERBEROS_PRINCIPAL_CONF_KEY = "hadoop.http.authentication.kerberos.principal"; > RBF: Security should use common AuthenticationFilter > > > Key: HDFS-14609 > URL: https://issues.apache.org/jira/browse/HDFS-14609 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: CR Hota >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-14609.001.patch, HDFS-14609.002.patch > > > We worked on router based federation security as part of HDFS-13532. We kept > it compatible with the way namenode works. However with HADOOP-16314 and > HDFS-16354 in trunk, auth filters seems to have been changed causing tests to > fail. > Changes are needed appropriately in RBF, mainly fixing broken tests. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14729) Upgrade Bootstrap and jQuery versions used in HDFS UIs
[ https://issues.apache.org/jira/browse/HDFS-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910549#comment-16910549 ] Sunil Govindan commented on HDFS-14729: --- +1 on latest patch. [~vivekratnavel] cud u pls submit patch to run jenkins > Upgrade Bootstrap and jQuery versions used in HDFS UIs > -- > > Key: HDFS-14729 > URL: https://issues.apache.org/jira/browse/HDFS-14729 > Project: Hadoop HDFS > Issue Type: Task > Components: ui >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > > The current versions of bootstrap and jquery have multiple medium severity > CVEs reported till date and needs to be updated to the latest versions with > no reported CVEs. > > I suggest updating the following libraries: > ||Library||From version||To version|| > |Bootstrap|3.3.7|3.4.1| > |jQuery|3.3.1|3.4.1| -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13101) Yet another fsimage corruption related to snapshot
[ https://issues.apache.org/jira/browse/HDFS-13101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-13101: --- Fix Version/s: 3.0.4 > Yet another fsimage corruption related to snapshot > -- > > Key: HDFS-13101 > URL: https://issues.apache.org/jira/browse/HDFS-13101 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Reporter: Yongjun Zhang >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3 > > Attachments: HDFS-13101.001.patch, HDFS-13101.002.patch, > HDFS-13101.003.patch, HDFS-13101.004.patch, HDFS-13101.branch-2.001.patch, > HDFS-13101.branch-2.8.patch, HDFS-13101.corruption_repro.patch, > HDFS-13101.corruption_repro_simplified.patch > > > Lately we saw case similar to HDFS-9406, even though HDFS-9406 fix is > present, so it's likely another case not covered by the fix. We are currently > trying to collect good fsimage + editlogs to replay to reproduce it and > investigate. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=297330=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297330 ] ASF GitHub Bot logged work on HDDS-1105: Author: ASF GitHub Bot Created on: 19/Aug/19 17:47 Start Date: 19/Aug/19 17:47 Worklog Time Spent: 10m Work Description: hanishakoneru commented on pull request #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297330) Time Spent: 3.5h (was: 3h 20m) > Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone > Manager. > > > Key: HDDS-1105 > URL: https://issues.apache.org/jira/browse/HDDS-1105 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > *Some context* > The FSCK server will periodically invoke this OM API passing in the most > recent sequence number of its own RocksDB instance. The OM will use the > RockDB getUpdateSince() API to answer this query. Since the getUpdateSince > API only works against the RocksDB WAL, we have to configure OM RocksDB WAL > (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log) with sufficient > max size to make this API useful. If the OM cannot get all transactions since > the given sequence number (due to WAL flushing), it can error out. In that > case the FSCK server can fall back to getting the entire checkpoint snapshot > implemented in HDDS-1085. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1988) Fix listParts API
Bharat Viswanadham created HDDS-1988: Summary: Fix listParts API Key: HDDS-1988 URL: https://issues.apache.org/jira/browse/HDDS-1988 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Bharat Viswanadham This Jira is to fix listBucket API in HA code path. In HA, we have an in-memory cache, where we put the result to in-memory cache and return the response, later it will be picked by double buffer thread and it will flush to disk. So, now when do listBuckets, it should use both in-memory cache and rocksdb bucket table to list buckets in a volume. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1989) Fix ApplyTransaction error handling in OzoneManagerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1989: - Parent: HDDS-505 Issue Type: Sub-task (was: Bug) > Fix ApplyTransaction error handling in OzoneManagerStateMachine > --- > > Key: HDDS-1989 > URL: https://issues.apache.org/jira/browse/HDDS-1989 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Priority: Major > > Right now, applyTransaction calls validateAndUpdateCache. This does not > return any error. We should check the OmResponse Status code, and see if it > is critical error we should completeExceptionally and stop ratis server. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1985) Fix listVolumes API
Bharat Viswanadham created HDDS-1985: Summary: Fix listVolumes API Key: HDDS-1985 URL: https://issues.apache.org/jira/browse/HDDS-1985 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Bharat Viswanadham This Jira is to fix listBucket API in HA code path. In HA, we have an in-memory cache, where we put the result to in-memory cache and return the response, later it will be picked by double buffer thread and it will flush to disk. So, now when do listBuckets, it should use both in-memory cache and rocksdb bucket table to list buckets in a volume. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1984) Fix listBucketApi
Bharat Viswanadham created HDDS-1984: Summary: Fix listBucketApi Key: HDDS-1984 URL: https://issues.apache.org/jira/browse/HDDS-1984 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham This Jira is to fix listBucket API in HA code path. In HA, we have an in-memory cache, where we put the result to in-memory cache and return the response, later it will be picked by double buffer thread and it will flush to disk. So, now when do listBuckets, it should use both in-memory cache and rocksdb bucket table to list buckets in a volume. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1984) Fix listBucketApi
[ https://issues.apache.org/jira/browse/HDDS-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1984: - Parent: HDDS-505 Issue Type: Sub-task (was: Bug) > Fix listBucketApi > - > > Key: HDDS-1984 > URL: https://issues.apache.org/jira/browse/HDDS-1984 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Priority: Major > > This Jira is to fix listBucket API in HA code path. > In HA, we have an in-memory cache, where we put the result to in-memory cache > and return the response, later it will be picked by double buffer thread and > it will flush to disk. So, now when do listBuckets, it should use both > in-memory cache and rocksdb bucket table to list buckets in a volume. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1990) Make expiry of Delegation tokens to OM HA model.
[ https://issues.apache.org/jira/browse/HDDS-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1990: - Parent: HDDS-505 Issue Type: Sub-task (was: Bug) > Make expiry of Delegation tokens to OM HA model. > > > Key: HDDS-1990 > URL: https://issues.apache.org/jira/browse/HDDS-1990 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Priority: Major > > Currently, when tokens expire, there is background thread which runs and > check if the token is expired, delete from in-memory map and DB. > In HA this deletion should go through ratis, to keep all OM's in sync. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1991) Remove RatisClient in OM HA
[ https://issues.apache.org/jira/browse/HDDS-1991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1991: - Parent: HDDS-505 Issue Type: Sub-task (was: Bug) > Remove RatisClient in OM HA > --- > > Key: HDDS-1991 > URL: https://issues.apache.org/jira/browse/HDDS-1991 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Priority: Major > > In OM, we use ratis server api's to submit request. We can remove the > RatisClient code from OM, which is no more used in submitting requests to > ratis. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1991) Remove RatisClient in OM HA
Bharat Viswanadham created HDDS-1991: Summary: Remove RatisClient in OM HA Key: HDDS-1991 URL: https://issues.apache.org/jira/browse/HDDS-1991 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham In OM, we use ratis server api's to submit request. We can remove the RatisClient code from OM, which is no more used in submitting requests to ratis. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1973) Implement OM RenewDelegationToken request to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng reassigned HDDS-1973: Assignee: Bharat Viswanadham (was: Siyao Meng) > Implement OM RenewDelegationToken request to use Cache and DoubleBuffer > --- > > Key: HDDS-1973 > URL: https://issues.apache.org/jira/browse/HDDS-1973 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > Implement OM RenewDelegationToken request to use OM Cache, double buffer. > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1200) Ozone Data Scrubbing : Checksum verification for chunks
[
https://issues.apache.org/jira/browse/HDDS-1200?focusedWorklogId=297436=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297436
]
ASF GitHub Bot logged work on HDDS-1200:
Author: ASF GitHub Bot
Created on: 19/Aug/19 21:44
Start Date: 19/Aug/19 21:44
Worklog Time Spent: 10m
Work Description: hgadre commented on pull request #1154: [HDDS-1200] Add
support for checksum verification in data scrubber
URL: https://github.com/apache/hadoop/pull/1154#discussion_r315425380
##
File path:
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainerCheck.java
##
@@ -120,10 +133,70 @@ public TestKeyValueContainerCheck(String metadataImpl) {
container.close();
// next run checks on a Closed Container
-valid = kvCheck.fullCheck();
+valid = kvCheck.fullCheck(new DataTransferThrottler(
+c.getBandwidthPerVolume()), null);
assertTrue(valid);
}
+ /**
+ * Sanity test, when there are corruptions induced.
+ * @throws Exception
+ */
+ @Test
+ public void testKeyValueContainerCheckCorruption() throws Exception {
+long containerID = 102;
+int deletedBlocks = 1;
+int normalBlocks = 3;
+int chunksPerBlock = 4;
+boolean valid = false;
+ContainerScrubberConfiguration sc = conf.getObject(
+ContainerScrubberConfiguration.class);
+
+// test Closed Container
+createContainerWithBlocks(containerID, normalBlocks, deletedBlocks, 65536,
+chunksPerBlock);
+File chunksPath = new File(containerData.getChunksPath());
+assertTrue(chunksPath.listFiles().length
+== (deletedBlocks + normalBlocks) * chunksPerBlock);
+
+container.close();
+
+KeyValueContainerCheck kvCheck =
+new KeyValueContainerCheck(containerData.getMetadataPath(), conf,
+containerID);
+
+File metaDir = new File(containerData.getMetadataPath());
+File dbFile = KeyValueContainerLocationUtil
+.getContainerDBFile(metaDir, containerID);
+containerData.setDbFile(dbFile);
+try(ReferenceCountedDB db =
+BlockUtils.getDB(containerData, conf);
+KeyValueBlockIterator kvIter = new KeyValueBlockIterator(containerID,
+new File(containerData.getContainerPath( {
+ BlockData block = kvIter.nextBlock();
+ assertTrue(!block.getChunks().isEmpty());
+ ContainerProtos.ChunkInfo c = block.getChunks().get(0);
+ File chunkFile = ChunkUtils.getChunkFile(containerData,
+ ChunkInfo.getFromProtoBuf(c));
+ long length = chunkFile.length();
+ assertTrue(length > 0);
+ // forcefully truncate the file to induce failure.
+ try (RandomAccessFile file = new RandomAccessFile(chunkFile, "rws")) {
+file.setLength(length / 2);
+ }
+ assertEquals(length/2, chunkFile.length());
+}
+
+// metadata check should pass.
+valid = kvCheck.fastCheck();
+assertTrue(valid);
+
+// checksum validation should fail.
+valid = kvCheck.fullCheck(new DataTransferThrottler(
+sc.getBandwidthPerVolume()), null);
+assertFalse(valid);
+ }
+
/**
* Creates a container with normal and deleted blocks.
* First it will insert normal blocks, and then it will insert
Review comment:
Not sure I am following you. Can you elaborate which part do you find
misleading? This function was present before this patch ...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297436)
Time Spent: 4.5h (was: 4h 20m)
> Ozone Data Scrubbing : Checksum verification for chunks
> ---
>
> Key: HDDS-1200
> URL: https://issues.apache.org/jira/browse/HDDS-1200
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Supratim Deka
>Assignee: Hrishikesh Gadre
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Background scrubber should read each chunk and verify the checksum.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1977) Fix checkstyle issues introduced by HDDS-1894
[
https://issues.apache.org/jira/browse/HDDS-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910800#comment-16910800
]
Bharat Viswanadham edited comment on HDDS-1977 at 8/19/19 10:30 PM:
This has been fixed by HDDS-1979. Closing this.
was (Author: bharatviswa):
This has been fixed by HDDS-1979.
> Fix checkstyle issues introduced by HDDS-1894
> -
>
> Key: HDDS-1977
> URL: https://issues.apache.org/jira/browse/HDDS-1977
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM Client
>Reporter: Nanda kumar
>Assignee: kevin su
>Priority: Major
> Labels: newbie
> Fix For: 0.5.0
>
> Attachments: HDDS-1977.001.patch, HDDS-1977.002.patch
>
>
> Fix the checkstyle issues introduced by HDDS-1894
> {noformat}
> [INFO] There are 6 errors reported by Checkstyle 8.8 with
> checkstyle/checkstyle.xml ruleset.
> [ERROR]
> src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ListPipelinesSubcommand.java:[41,23]
> (whitespace) ParenPad: '(' is followed by whitespace.
> [ERROR]
> src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ListPipelinesSubcommand.java:[42]
> (sizes) LineLength: Line is longer than 80 characters (found 88).
> [ERROR]
> src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ListPipelinesSubcommand.java:[46,23]
> (whitespace) ParenPad: '(' is followed by whitespace.
> [ERROR]
> src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ListPipelinesSubcommand.java:[47]
> (sizes) LineLength: Line is longer than 80 characters (found 90).
> [ERROR]
> src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ListPipelinesSubcommand.java:[59]
> (sizes) LineLength: Line is longer than 80 characters (found 116).
> [ERROR]
> src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ListPipelinesSubcommand.java:[60]
> (sizes) LineLength: Line is longer than 80 characters (found 120).
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1977) Fix checkstyle issues introduced by HDDS-1894
[
https://issues.apache.org/jira/browse/HDDS-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-1977:
-
Fix Version/s: 0.5.0
Resolution: Duplicate
Status: Resolved (was: Patch Available)
This has been fixed by HDDS-1979.
> Fix checkstyle issues introduced by HDDS-1894
> -
>
> Key: HDDS-1977
> URL: https://issues.apache.org/jira/browse/HDDS-1977
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM Client
>Reporter: Nanda kumar
>Assignee: kevin su
>Priority: Major
> Labels: newbie
> Fix For: 0.5.0
>
> Attachments: HDDS-1977.001.patch, HDDS-1977.002.patch
>
>
> Fix the checkstyle issues introduced by HDDS-1894
> {noformat}
> [INFO] There are 6 errors reported by Checkstyle 8.8 with
> checkstyle/checkstyle.xml ruleset.
> [ERROR]
> src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ListPipelinesSubcommand.java:[41,23]
> (whitespace) ParenPad: '(' is followed by whitespace.
> [ERROR]
> src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ListPipelinesSubcommand.java:[42]
> (sizes) LineLength: Line is longer than 80 characters (found 88).
> [ERROR]
> src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ListPipelinesSubcommand.java:[46,23]
> (whitespace) ParenPad: '(' is followed by whitespace.
> [ERROR]
> src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ListPipelinesSubcommand.java:[47]
> (sizes) LineLength: Line is longer than 80 characters (found 90).
> [ERROR]
> src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ListPipelinesSubcommand.java:[59]
> (sizes) LineLength: Line is longer than 80 characters (found 116).
> [ERROR]
> src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ListPipelinesSubcommand.java:[60]
> (sizes) LineLength: Line is longer than 80 characters (found 120).
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-14208) A large number missingblocks happend after failover to active.
[ https://issues.apache.org/jira/browse/HDFS-14208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14208. Resolution: Duplicate Thanks [~xuzq_zander] and [~hexiaoqiao] > A large number missingblocks happend after failover to active. > -- > > Key: HDFS-14208 > URL: https://issues.apache.org/jira/browse/HDFS-14208 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.0.2, 3.1.1, 3.0.3 >Reporter: xuzq >Priority: Major > > When cluster is very large, Standby startup takes about 1 hours. After > Standby starts and exits SafeMode, a large number of missingblocks often > appeared after failover from standby to active. And the missingblocks will > gradually disappear after 6 hours(block report interval). > According the log and find only half the blocks of one DataNode be processed > in SaftMode. > > Because the lease expired after processed some blocks of one DataNode. > [MergeRequest|https://github.com/apache/hadoop/pull/467] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1927) Consolidate add/remove Acl into OzoneAclUtil class
[
https://issues.apache.org/jira/browse/HDDS-1927?focusedWorklogId=297490=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297490
]
ASF GitHub Bot logged work on HDDS-1927:
Author: ASF GitHub Bot
Created on: 19/Aug/19 23:24
Start Date: 19/Aug/19 23:24
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1263:
HDDS-1927. Consolidate add/remove Acl into OzoneAclUtil class. Contri…
URL: https://github.com/apache/hadoop/pull/1263#discussion_r315452170
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java
##
@@ -235,123 +231,22 @@ public FileEncryptionInfo getFileEncryptionInfo() {
return encInfo;
}
- public List getAcls() {
+ public List getAcls() {
Review comment:
What I mean here is:
Take an example OMBucketAddAclRequest
```
public OMBucketAddAclRequest(OMRequest omRequest) {
super(omRequest, bucketAddAclOp);
OzoneManagerProtocolProtos.AddAclRequest addAclRequest =
getOmRequest().getAddAclRequest();
path = addAclRequest.getObj().getPath();
ozoneAcls = Lists.newArrayList(
OzoneAcl.fromProtobuf(addAclRequest.getAcl()));
```
If OMBucketInfo, has direct protobuf structures we can avoid the conversion.
Now, we do 1 conversion in constructor, and the other when write to DB.
ozoneAcl -> proto -> byte Array. If we can have direct protobuf structures, we
can save unnecessary protobuf conversions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297490)
Time Spent: 2h 50m (was: 2h 40m)
> Consolidate add/remove Acl into OzoneAclUtil class
> --
>
> Key: HDDS-1927
> URL: https://issues.apache.org/jira/browse/HDDS-1927
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> This Jira is created based on @xiaoyu comment on HDDS-1884
> Can we abstract these add/remove logic into common AclUtil class as we can
> see similar logic in both bucket manager and key manager? For example,
> public static boolean addAcl(List existingAcls, OzoneAcl newAcl)
> public static boolean removeAcl(List existingAcls, OzoneAcl newAcl)
>
> But to do this, we need both OmKeyInfo and OMBucketInfo to use list of
> OzoneAcl/OzoneAclInfo.
> This Jira is to do that refactor, and also address above comment to move
> common logic to AclUtils.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1927) Consolidate add/remove Acl into OzoneAclUtil class
[
https://issues.apache.org/jira/browse/HDDS-1927?focusedWorklogId=297491=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297491
]
ASF GitHub Bot logged work on HDDS-1927:
Author: ASF GitHub Bot
Created on: 19/Aug/19 23:25
Start Date: 19/Aug/19 23:25
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1263:
HDDS-1927. Consolidate add/remove Acl into OzoneAclUtil class. Contri…
URL: https://github.com/apache/hadoop/pull/1263#discussion_r315452344
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java
##
@@ -235,123 +231,22 @@ public FileEncryptionInfo getFileEncryptionInfo() {
return encInfo;
}
- public List getAcls() {
+ public List getAcls() {
Review comment:
> As a result, we will only do the protobuf conversion in those places.
Not sure if I am missing something here. From the above example, we are
doing the conversion in 2 places.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297491)
Time Spent: 3h (was: 2h 50m)
> Consolidate add/remove Acl into OzoneAclUtil class
> --
>
> Key: HDDS-1927
> URL: https://issues.apache.org/jira/browse/HDDS-1927
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> This Jira is created based on @xiaoyu comment on HDDS-1884
> Can we abstract these add/remove logic into common AclUtil class as we can
> see similar logic in both bucket manager and key manager? For example,
> public static boolean addAcl(List existingAcls, OzoneAcl newAcl)
> public static boolean removeAcl(List existingAcls, OzoneAcl newAcl)
>
> But to do this, we need both OmKeyInfo and OMBucketInfo to use list of
> OzoneAcl/OzoneAclInfo.
> This Jira is to do that refactor, and also address above comment to move
> common logic to AclUtils.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14648) DeadNodeDetector state machine model
[ https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910858#comment-16910858 ] Lisheng Sun commented on HDFS-14648: hi [~linyiqun] [~jojochuang], [~hexiaoqiao] [~xkrogen] [~elgoiri] Could you have time to review this patch? Thank you. > DeadNodeDetector state machine model > > > Key: HDFS-14648 > URL: https://issues.apache.org/jira/browse/HDFS-14648 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14648.001.patch, HDFS-14648.002.patch, > HDFS-14648.003.patch > > > This Jira constructs DeadNodeDetector state machine model. The function it > implements as follow: > # After DFSInputstream detects some DataNode die, it put in DeadNodeDetector > and share this information to others in the same DFSClient. The ohter > DFSInputstreams will not read this DataNode. > # DeadNodeDetector also have DFSInputstream reference relationships to each > DataNode. When DFSInputstream close, DeadNodeDetector also remove this > reference. If some DeadNode of DeadNodeDetector is not read by > DFSInputstream, it also is removed from DeadNodeDetector. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1927) Consolidate add/remove Acl into OzoneAclUtil class
[
https://issues.apache.org/jira/browse/HDDS-1927?focusedWorklogId=297516=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297516
]
ASF GitHub Bot logged work on HDDS-1927:
Author: ASF GitHub Bot
Created on: 20/Aug/19 00:46
Start Date: 20/Aug/19 00:46
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1263:
HDDS-1927. Consolidate add/remove Acl into OzoneAclUtil class. Contri…
URL: https://github.com/apache/hadoop/pull/1263#discussion_r315467845
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java
##
@@ -235,123 +231,22 @@ public FileEncryptionInfo getFileEncryptionInfo() {
return encInfo;
}
- public List getAcls() {
+ public List getAcls() {
Review comment:
`Raw protobuf class can't be extended easily for acl operations. We will
have to to wrap the acl operations anyway.`
Not understood what do you mean by wrap ACL operations anyway.
`Given the scatter logic of ACL ops for volume/bucket/key/prefix, I would
prefer to have a unified code base that can be maintained/tested easily in the
long run.`
I understand that instead of spreading logic across in multiple places keep
acl logic in single place, which can be used in all bucket/key/prefix ops. I
agree with that point. But I have not understood why same cannot be done with
protobuf acl objects.
3. Agreed thar write acl operations/ will be lesser than read Acls.
But if you think this is the right way to go, I am fine with it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297516)
Time Spent: 3h 20m (was: 3h 10m)
> Consolidate add/remove Acl into OzoneAclUtil class
> --
>
> Key: HDDS-1927
> URL: https://issues.apache.org/jira/browse/HDDS-1927
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> This Jira is created based on @xiaoyu comment on HDDS-1884
> Can we abstract these add/remove logic into common AclUtil class as we can
> see similar logic in both bucket manager and key manager? For example,
> public static boolean addAcl(List existingAcls, OzoneAcl newAcl)
> public static boolean removeAcl(List existingAcls, OzoneAcl newAcl)
>
> But to do this, we need both OmKeyInfo and OMBucketInfo to use list of
> OzoneAcl/OzoneAclInfo.
> This Jira is to do that refactor, and also address above comment to move
> common logic to AclUtils.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1986) Fix listkeys API
[ https://issues.apache.org/jira/browse/HDDS-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1986: - Description: This Jira is to fix listKeys API in HA code path. In HA, we have an in-memory cache, where we put the result to in-memory cache and return the response, later it will be picked by double buffer thread and it will flush to disk. So, now when do listkeys, it should use both in-memory cache and rocksdb key table to list keys in a bucket. was: This Jira is to fix listBucket API in HA code path. In HA, we have an in-memory cache, where we put the result to in-memory cache and return the response, later it will be picked by double buffer thread and it will flush to disk. So, now when do listBuckets, it should use both in-memory cache and rocksdb bucket table to list buckets in a volume. > Fix listkeys API > > > Key: HDDS-1986 > URL: https://issues.apache.org/jira/browse/HDDS-1986 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Priority: Major > > This Jira is to fix listKeys API in HA code path. > In HA, we have an in-memory cache, where we put the result to in-memory cache > and return the response, later it will be picked by double buffer thread and > it will flush to disk. So, now when do listkeys, it should use both in-memory > cache and rocksdb key table to list keys in a bucket. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1987) Fix listStatus API
Bharat Viswanadham created HDDS-1987: Summary: Fix listStatus API Key: HDDS-1987 URL: https://issues.apache.org/jira/browse/HDDS-1987 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Bharat Viswanadham This Jira is to fix listKeys API in HA code path. In HA, we have an in-memory cache, where we put the result to in-memory cache and return the response, later it will be picked by double buffer thread and it will flush to disk. So, now when do listkeys, it should use both in-memory cache and rocksdb key table to list keys in a bucket. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1989) Fix ApplyTransaction error handling in OzoneManagerStateMachine
Bharat Viswanadham created HDDS-1989: Summary: Fix ApplyTransaction error handling in OzoneManagerStateMachine Key: HDDS-1989 URL: https://issues.apache.org/jira/browse/HDDS-1989 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Right now, applyTransaction calls validateAndUpdateCache. This does not return any error. We should check the OmResponse Status code, and see if it is critical error we should completeExceptionally and stop ratis server. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14678) Allow triggerBlockReport to a specific namenode
[ https://issues.apache.org/jira/browse/HDFS-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910767#comment-16910767 ] Leon Gao commented on HDFS-14678: - Yes we need this in branch 2, I will create another PR for this. > Allow triggerBlockReport to a specific namenode > --- > > Key: HDFS-14678 > URL: https://issues.apache.org/jira/browse/HDFS-14678 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.8.2 >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Major > Fix For: 3.3.0 > > > In our largest prod cluster (running 2.8.2) we have >3k hosts. Every time > when rolling restarting NNs we will need to wait for block report which takes > >2.5 hours for each NN. > One way to make it faster is to manually trigger a full block report from all > datanodes. [HDFS-7278|https://issues.apache.org/jira/browse/HDFS-7278]. > However, the current triggerBlockReport command will trigger a block report > on all NNs which will flood the active NN as well. > A quick solution will be adding an option to specify a NN that the manually > triggered block report will go to, something like: > *_hdfs dfsadmin [-triggerBlockReport [-incremental] ] > [-namenode] _* > So when doing a restart of standby NN or observer NN we can trigger an > aggressive block report to a specific NN to exit safemode faster without > risking active NN performance. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1975) Implement default acls for bucket/volume/key for OM HA code
[ https://issues.apache.org/jira/browse/HDDS-1975?focusedWorklogId=297431=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297431 ] ASF GitHub Bot logged work on HDDS-1975: Author: ASF GitHub Bot Created on: 19/Aug/19 21:33 Start Date: 19/Aug/19 21:33 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #1315: HDDS-1975. Implement default acls for bucket/volume/key for OM HA code. URL: https://github.com/apache/hadoop/pull/1315#issuecomment-522762718 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297431) Time Spent: 20m (was: 10m) > Implement default acls for bucket/volume/key for OM HA code > --- > > Key: HDDS-1975 > URL: https://issues.apache.org/jira/browse/HDDS-1975 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > This Jira is to implement default ACLs for Ozone volume/bucket/key. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14735) File could only be replicated to 0 nodes instead of minReplication (=1)
[ https://issues.apache.org/jira/browse/HDFS-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910854#comment-16910854 ] Tatyana Alexeyev commented on HDFS-14735: - Hello Chen, I have enabled the debugger... What is the next step? Thanks, tanya > File could only be replicated to 0 nodes instead of minReplication (=1) > --- > > Key: HDFS-14735 > URL: https://issues.apache.org/jira/browse/HDFS-14735 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Tatyana Alexeyev >Priority: Major > > Hello I have intermitent error when running my EMR Hadoop Cluster: > "Error: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /user/sphdadm/_sqoop/00501bd7b05e4182b5006b9d51 > bafb7f_f405b2f3/_temporary/1/_temporary/attempt_1565136887564_20057_m_00_0/part-m-0.snappy > could only be replicated to 0 nodes instead of minReplication (=1). There > are 5 datanode(s) running and no node(s) are excluded in this operation." > I am running Hadoop version > sphdadm@ip-10-6-15-108 hadoop]$ hadoop version > Hadoop 2.8.5-amzn-4 > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1975) Implement default acls for bucket/volume/key for OM HA code
[ https://issues.apache.org/jira/browse/HDDS-1975?focusedWorklogId=297505=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297505 ] ASF GitHub Bot logged work on HDDS-1975: Author: ASF GitHub Bot Created on: 20/Aug/19 00:05 Start Date: 20/Aug/19 00:05 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1315: HDDS-1975. Implement default acls for bucket/volume/key for OM HA code. URL: https://github.com/apache/hadoop/pull/1315#issuecomment-522799727 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1143 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 30 | Maven dependency ordering for branch | | +1 | mvninstall | 630 | trunk passed | | +1 | compile | 388 | trunk passed | | +1 | checkstyle | 71 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 968 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 173 | trunk passed | | 0 | spotbugs | 458 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 678 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 20 | Maven dependency ordering for patch | | +1 | mvninstall | 585 | the patch passed | | +1 | compile | 399 | the patch passed | | +1 | javac | 399 | the patch passed | | +1 | checkstyle | 90 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 758 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 176 | the patch passed | | +1 | findbugs | 708 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 379 | hadoop-hdds in the patch passed. | | -1 | unit | 333 | hadoop-ozone in the patch failed. | | +1 | asflicense | 56 | The patch does not generate ASF License warnings. | | | | 7750 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.request.s3.bucket.TestS3BucketCreateRequest | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1315/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1315 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux a9c39b494fac 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 4f925af | | Default Java | 1.8.0_222 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1315/3/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1315/3/testReport/ | | Max. process+thread count | 1325 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1315/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297505) Time Spent: 40m (was: 0.5h) > Implement default acls for bucket/volume/key for OM HA code > --- > > Key: HDDS-1975 > URL: https://issues.apache.org/jira/browse/HDDS-1975 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > This Jira is to implement default ACLs for Ozone volume/bucket/key. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (HDDS-1927) Consolidate add/remove Acl into OzoneAclUtil class
[
https://issues.apache.org/jira/browse/HDDS-1927?focusedWorklogId=297519=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297519
]
ASF GitHub Bot logged work on HDDS-1927:
Author: ASF GitHub Bot
Created on: 20/Aug/19 00:54
Start Date: 20/Aug/19 00:54
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1263:
HDDS-1927. Consolidate add/remove Acl into OzoneAclUtil class. Contri…
URL: https://github.com/apache/hadoop/pull/1263#discussion_r315468225
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java
##
@@ -235,123 +231,22 @@ public FileEncryptionInfo getFileEncryptionInfo() {
return encInfo;
}
- public List getAcls() {
+ public List getAcls() {
Review comment:
And also this will not only help acls But bucket/key creation also, as now
Bucket/KeyInfo will have proto buf -> internal Ozone object. This also can be
avoided. (So, when each key creation we don't need to convert acls set during
the creation of key from proto to OzoneAcl Objects.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297519)
Time Spent: 3h 40m (was: 3.5h)
> Consolidate add/remove Acl into OzoneAclUtil class
> --
>
> Key: HDDS-1927
> URL: https://issues.apache.org/jira/browse/HDDS-1927
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> This Jira is created based on @xiaoyu comment on HDDS-1884
> Can we abstract these add/remove logic into common AclUtil class as we can
> see similar logic in both bucket manager and key manager? For example,
> public static boolean addAcl(List existingAcls, OzoneAcl newAcl)
> public static boolean removeAcl(List existingAcls, OzoneAcl newAcl)
>
> But to do this, we need both OmKeyInfo and OMBucketInfo to use list of
> OzoneAcl/OzoneAclInfo.
> This Jira is to do that refactor, and also address above comment to move
> common logic to AclUtils.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1975) Implement default acls for bucket/volume/key for OM HA code
[ https://issues.apache.org/jira/browse/HDDS-1975?focusedWorklogId=297532=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297532 ] ASF GitHub Bot logged work on HDDS-1975: Author: ASF GitHub Bot Created on: 20/Aug/19 01:25 Start Date: 20/Aug/19 01:25 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #1315: HDDS-1975. Implement default acls for bucket/volume/key for OM HA code. URL: https://github.com/apache/hadoop/pull/1315#issuecomment-522814418 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297532) Time Spent: 1h (was: 50m) > Implement default acls for bucket/volume/key for OM HA code > --- > > Key: HDDS-1975 > URL: https://issues.apache.org/jira/browse/HDDS-1975 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > This Jira is to implement default ACLs for Ozone volume/bucket/key. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1987) Fix listStatus API
[ https://issues.apache.org/jira/browse/HDDS-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1987: - Description: This Jira is to fix listKeys API in HA code path. In HA, we have an in-memory cache, where we put the result to in-memory cache and return the response, later it will be picked by double buffer thread and it will flush to disk. So, now when do listStatus, it should use both in-memory cache and rocksdb key table to list Status of Keys in a bucket. was: This Jira is to fix listKeys API in HA code path. In HA, we have an in-memory cache, where we put the result to in-memory cache and return the response, later it will be picked by double buffer thread and it will flush to disk. So, now when do listkeys, it should use both in-memory cache and rocksdb key table to list keys in a bucket. > Fix listStatus API > -- > > Key: HDDS-1987 > URL: https://issues.apache.org/jira/browse/HDDS-1987 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Priority: Major > > This Jira is to fix listKeys API in HA code path. > In HA, we have an in-memory cache, where we put the result to in-memory cache > and return the response, later it will be picked by double buffer thread and > it will flush to disk. So, now when do listStatus, it should use both > in-memory cache and rocksdb key table to list Status of Keys in a bucket. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1988) Fix listParts API
[ https://issues.apache.org/jira/browse/HDDS-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1988: - Description: This Jira is to fix listBucket API in HA code path. In HA, we have an in-memory cache, where we put the result to in-memory cache and return the response, later it will be picked by double buffer thread and it will flush to disk. So, now when do listParts of a MPU key, it should use both in-memory cache and rocksdb mpu table to list parts of a mpu key. was: This Jira is to fix listBucket API in HA code path. In HA, we have an in-memory cache, where we put the result to in-memory cache and return the response, later it will be picked by double buffer thread and it will flush to disk. So, now when do listBuckets, it should use both in-memory cache and rocksdb bucket table to list buckets in a volume. > Fix listParts API > - > > Key: HDDS-1988 > URL: https://issues.apache.org/jira/browse/HDDS-1988 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Priority: Major > > This Jira is to fix listBucket API in HA code path. > In HA, we have an in-memory cache, where we put the result to in-memory cache > and return the response, later it will be picked by double buffer thread and > it will flush to disk. So, now when do listParts of a MPU key, it should use > both in-memory cache and rocksdb mpu table to list parts of a mpu key. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1990) Make expiry of Delegation tokens to OM HA model.
Bharat Viswanadham created HDDS-1990: Summary: Make expiry of Delegation tokens to OM HA model. Key: HDDS-1990 URL: https://issues.apache.org/jira/browse/HDDS-1990 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Currently, when tokens expire, there is background thread which runs and check if the token is expired, delete from in-memory map and DB. In HA this deletion should go through ratis, to keep all OM's in sync. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1992) Have a separate CacheTable class for cache based tables.
[
https://issues.apache.org/jira/browse/HDDS-1992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-1992:
-
Priority: Major (was: Blocker)
> Have a separate CacheTable class for cache based tables.
>
>
> Key: HDDS-1992
> URL: https://issues.apache.org/jira/browse/HDDS-1992
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: om
>Reporter: Bharat Viswanadham
>Priority: Major
> Fix For: 0.4.1
>
>
> HDDS-1499 introduced a new caching layer together with a double-buffer based
> db writer to support OM HA.
> TLDR: I think the caching layer is not updated for new volume creation. And
> (slightly related to this problem) I suggest to separated the TypedTable and
> the caching layer.
> ## How to reproduce the problem?
> 1. Start a docker compose cluster
> 2. Create one volume (let's say `/vol1`)
> 3. Restart the om (!)
> 4. Try to create an _other_ volume twice!
> ```
> bash-4.2$ ozone sh volume create /vol2
> 2019-08-07 12:29:47 INFO RpcClient:288 - Creating Volume: vol2, with hadoop
> as owner.
> bash-4.2$ ozone sh volume create /vol2
> 2019-08-07 12:29:50 INFO RpcClient:288 - Creating Volume: vol2, with hadoop
> as owner.
> ```
> Expected behavior is an error:
> {code}
> bash-4.2$ ozone sh volume create /vol1
> 2019-08-07 09:48:39 INFO RpcClient:288 - Creating Volume: vol1, with hadoop
> as owner.
> bash-4.2$ ozone sh volume create /vol1
> 2019-08-07 09:48:42 INFO RpcClient:288 - Creating Volume: vol1, with hadoop
> as owner.
> VOLUME_ALREADY_EXISTS
> {code}
> The problem is that the new cache is used even for the old code path
> (TypedTable):
> {code}
> @Override
> public VALUE get(KEY key) throws IOException {
> // Here the metadata lock will guarantee that cache is not updated for
> same
> // key during get key.
> CacheResult> cacheResult =
> cache.lookup(new CacheKey<>(key));
> if (cacheResult.getCacheStatus() == EXISTS) {
> return cacheResult.getValue().getCacheValue();
> } else if (cacheResult.getCacheStatus() == NOT_EXIST) {
> return null;
> } else {
> return getFromTable(key);
> }
> }
> {code}
> For volume table after the FIRST start it always returns with
> `getFromTable(key)` due to the condition in the `TableCacheImpl.lookup`:
> {code}
> public CacheResult lookup(CACHEKEY cachekey) {
> if (cache.size() == 0) {
> return new CacheResult<>(CacheResult.CacheStatus.MAY_EXIST,
> null);
> }
> {code}
> But after a restart the cache is pre-loaded by the TypedTable.constructor.
> After the restart, the real caching logic will be used (as cache.size()>0),
> which cause a problem as the cache is NOT updated from the old code path.
> An additional problem is that the cache is turned on for all the metadata
> table even if the cache is not required...
> ## Proposed solution
> As I commented at HDDS-1499 this caching layer is not a "traditional cache".
> It's not updated during the typedTable.put() call but updated by a separated
> component during double-buffer flash.
> I would suggest to remove the cache related methods from TypedTable (move to
> a separated implementation). I think this kind of caching can be independent
> from the TypedTable implementation. We can continue to use the simple
> TypedTable everywhere where we don't need to use any kind of caching.
> For caching we can use a separated object. It would make it more visible that
> the cache should always be updated manually all the time. This separated
> caching utility may include a reference to the original TypedTable/Table.
> With this approach we can separate the different responsibilities but provide
> the same functionality.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1992) Have a separate CacheTable class for cache based tables.
[
https://issues.apache.org/jira/browse/HDDS-1992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-1992:
-
Parent: HDDS-505
Issue Type: Sub-task (was: Bug)
> Have a separate CacheTable class for cache based tables.
>
>
> Key: HDDS-1992
> URL: https://issues.apache.org/jira/browse/HDDS-1992
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: om
>Reporter: Bharat Viswanadham
>Priority: Major
> Fix For: 0.4.1
>
>
> HDDS-1499 introduced a new caching layer together with a double-buffer based
> db writer to support OM HA.
> TLDR: I think the caching layer is not updated for new volume creation. And
> (slightly related to this problem) I suggest to separated the TypedTable and
> the caching layer.
> ## How to reproduce the problem?
> 1. Start a docker compose cluster
> 2. Create one volume (let's say `/vol1`)
> 3. Restart the om (!)
> 4. Try to create an _other_ volume twice!
> ```
> bash-4.2$ ozone sh volume create /vol2
> 2019-08-07 12:29:47 INFO RpcClient:288 - Creating Volume: vol2, with hadoop
> as owner.
> bash-4.2$ ozone sh volume create /vol2
> 2019-08-07 12:29:50 INFO RpcClient:288 - Creating Volume: vol2, with hadoop
> as owner.
> ```
> Expected behavior is an error:
> {code}
> bash-4.2$ ozone sh volume create /vol1
> 2019-08-07 09:48:39 INFO RpcClient:288 - Creating Volume: vol1, with hadoop
> as owner.
> bash-4.2$ ozone sh volume create /vol1
> 2019-08-07 09:48:42 INFO RpcClient:288 - Creating Volume: vol1, with hadoop
> as owner.
> VOLUME_ALREADY_EXISTS
> {code}
> The problem is that the new cache is used even for the old code path
> (TypedTable):
> {code}
> @Override
> public VALUE get(KEY key) throws IOException {
> // Here the metadata lock will guarantee that cache is not updated for
> same
> // key during get key.
> CacheResult> cacheResult =
> cache.lookup(new CacheKey<>(key));
> if (cacheResult.getCacheStatus() == EXISTS) {
> return cacheResult.getValue().getCacheValue();
> } else if (cacheResult.getCacheStatus() == NOT_EXIST) {
> return null;
> } else {
> return getFromTable(key);
> }
> }
> {code}
> For volume table after the FIRST start it always returns with
> `getFromTable(key)` due to the condition in the `TableCacheImpl.lookup`:
> {code}
> public CacheResult lookup(CACHEKEY cachekey) {
> if (cache.size() == 0) {
> return new CacheResult<>(CacheResult.CacheStatus.MAY_EXIST,
> null);
> }
> {code}
> But after a restart the cache is pre-loaded by the TypedTable.constructor.
> After the restart, the real caching logic will be used (as cache.size()>0),
> which cause a problem as the cache is NOT updated from the old code path.
> An additional problem is that the cache is turned on for all the metadata
> table even if the cache is not required...
> ## Proposed solution
> As I commented at HDDS-1499 this caching layer is not a "traditional cache".
> It's not updated during the typedTable.put() call but updated by a separated
> component during double-buffer flash.
> I would suggest to remove the cache related methods from TypedTable (move to
> a separated implementation). I think this kind of caching can be independent
> from the TypedTable implementation. We can continue to use the simple
> TypedTable everywhere where we don't need to use any kind of caching.
> For caching we can use a separated object. It would make it more visible that
> the cache should always be updated manually all the time. This separated
> caching utility may include a reference to the original TypedTable/Table.
> With this approach we can separate the different responsibilities but provide
> the same functionality.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1992) Have a separate CacheTable class for cache based tables.
[
https://issues.apache.org/jira/browse/HDDS-1992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham reassigned HDDS-1992:
Assignee: (was: Bharat Viswanadham)
> Have a separate CacheTable class for cache based tables.
>
>
> Key: HDDS-1992
> URL: https://issues.apache.org/jira/browse/HDDS-1992
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: om
>Reporter: Bharat Viswanadham
>Priority: Blocker
> Fix For: 0.4.1
>
>
> HDDS-1499 introduced a new caching layer together with a double-buffer based
> db writer to support OM HA.
> TLDR: I think the caching layer is not updated for new volume creation. And
> (slightly related to this problem) I suggest to separated the TypedTable and
> the caching layer.
> ## How to reproduce the problem?
> 1. Start a docker compose cluster
> 2. Create one volume (let's say `/vol1`)
> 3. Restart the om (!)
> 4. Try to create an _other_ volume twice!
> ```
> bash-4.2$ ozone sh volume create /vol2
> 2019-08-07 12:29:47 INFO RpcClient:288 - Creating Volume: vol2, with hadoop
> as owner.
> bash-4.2$ ozone sh volume create /vol2
> 2019-08-07 12:29:50 INFO RpcClient:288 - Creating Volume: vol2, with hadoop
> as owner.
> ```
> Expected behavior is an error:
> {code}
> bash-4.2$ ozone sh volume create /vol1
> 2019-08-07 09:48:39 INFO RpcClient:288 - Creating Volume: vol1, with hadoop
> as owner.
> bash-4.2$ ozone sh volume create /vol1
> 2019-08-07 09:48:42 INFO RpcClient:288 - Creating Volume: vol1, with hadoop
> as owner.
> VOLUME_ALREADY_EXISTS
> {code}
> The problem is that the new cache is used even for the old code path
> (TypedTable):
> {code}
> @Override
> public VALUE get(KEY key) throws IOException {
> // Here the metadata lock will guarantee that cache is not updated for
> same
> // key during get key.
> CacheResult> cacheResult =
> cache.lookup(new CacheKey<>(key));
> if (cacheResult.getCacheStatus() == EXISTS) {
> return cacheResult.getValue().getCacheValue();
> } else if (cacheResult.getCacheStatus() == NOT_EXIST) {
> return null;
> } else {
> return getFromTable(key);
> }
> }
> {code}
> For volume table after the FIRST start it always returns with
> `getFromTable(key)` due to the condition in the `TableCacheImpl.lookup`:
> {code}
> public CacheResult lookup(CACHEKEY cachekey) {
> if (cache.size() == 0) {
> return new CacheResult<>(CacheResult.CacheStatus.MAY_EXIST,
> null);
> }
> {code}
> But after a restart the cache is pre-loaded by the TypedTable.constructor.
> After the restart, the real caching logic will be used (as cache.size()>0),
> which cause a problem as the cache is NOT updated from the old code path.
> An additional problem is that the cache is turned on for all the metadata
> table even if the cache is not required...
> ## Proposed solution
> As I commented at HDDS-1499 this caching layer is not a "traditional cache".
> It's not updated during the typedTable.put() call but updated by a separated
> component during double-buffer flash.
> I would suggest to remove the cache related methods from TypedTable (move to
> a separated implementation). I think this kind of caching can be independent
> from the TypedTable implementation. We can continue to use the simple
> TypedTable everywhere where we don't need to use any kind of caching.
> For caching we can use a separated object. It would make it more visible that
> the cache should always be updated manually all the time. This separated
> caching utility may include a reference to the original TypedTable/Table.
> With this approach we can separate the different responsibilities but provide
> the same functionality.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1984) Fix listBucket API
[ https://issues.apache.org/jira/browse/HDDS-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1984: - Summary: Fix listBucket API (was: Fix listBucketApi) > Fix listBucket API > -- > > Key: HDDS-1984 > URL: https://issues.apache.org/jira/browse/HDDS-1984 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Priority: Major > > This Jira is to fix listBucket API in HA code path. > In HA, we have an in-memory cache, where we put the result to in-memory cache > and return the response, later it will be picked by double buffer thread and > it will flush to disk. So, now when do listBuckets, it should use both > in-memory cache and rocksdb bucket table to list buckets in a volume. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1993) Merge OzoneManagerRequestHandler and OzoneManagerHARequestHandlerImpl
[ https://issues.apache.org/jira/browse/HDDS-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1993: - Parent: HDDS-505 Issue Type: Sub-task (was: Bug) > Merge OzoneManagerRequestHandler and OzoneManagerHARequestHandlerImpl > - > > Key: HDDS-1993 > URL: https://issues.apache.org/jira/browse/HDDS-1993 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Priority: Major > > Once HA and Non-HA code are merged to use newly OM HA code. We can merge > these classes, and remove the unused code. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1993) Merge OzoneManagerRequestHandler and OzoneManagerHARequestHandlerImpl
Bharat Viswanadham created HDDS-1993: Summary: Merge OzoneManagerRequestHandler and OzoneManagerHARequestHandlerImpl Key: HDDS-1993 URL: https://issues.apache.org/jira/browse/HDDS-1993 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Once HA and Non-HA code are merged to use newly OM HA code. We can merge these classes, and remove the unused code. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1927) Consolidate add/remove Acl into OzoneAclUtil class
[
https://issues.apache.org/jira/browse/HDDS-1927?focusedWorklogId=297478=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297478
]
ASF GitHub Bot logged work on HDDS-1927:
Author: ASF GitHub Bot
Created on: 19/Aug/19 22:34
Start Date: 19/Aug/19 22:34
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1263:
HDDS-1927. Consolidate add/remove Acl into OzoneAclUtil class. Contri…
URL: https://github.com/apache/hadoop/pull/1263#discussion_r315440404
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java
##
@@ -235,123 +231,22 @@ public FileEncryptionInfo getFileEncryptionInfo() {
return encInfo;
}
- public List getAcls() {
+ public List getAcls() {
Review comment:
We can leave this as OzoneAclInfo, and make OmBucketInfo OzoneAcl ->
OzoneAclInfo.
In this way, we can avoid conversion of OzoneAcl list during OM operations.
And this will be useful in HA code path, as we operate directly OMRequest
objects. (I mean here we get the data from OMRequest, and perform operation and
store in cache)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297478)
Time Spent: 2h 20m (was: 2h 10m)
> Consolidate add/remove Acl into OzoneAclUtil class
> --
>
> Key: HDDS-1927
> URL: https://issues.apache.org/jira/browse/HDDS-1927
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> This Jira is created based on @xiaoyu comment on HDDS-1884
> Can we abstract these add/remove logic into common AclUtil class as we can
> see similar logic in both bucket manager and key manager? For example,
> public static boolean addAcl(List existingAcls, OzoneAcl newAcl)
> public static boolean removeAcl(List existingAcls, OzoneAcl newAcl)
>
> But to do this, we need both OmKeyInfo and OMBucketInfo to use list of
> OzoneAcl/OzoneAclInfo.
> This Jira is to do that refactor, and also address above comment to move
> common logic to AclUtils.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1927) Consolidate add/remove Acl into OzoneAclUtil class
[
https://issues.apache.org/jira/browse/HDDS-1927?focusedWorklogId=297479=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297479
]
ASF GitHub Bot logged work on HDDS-1927:
Author: ASF GitHub Bot
Created on: 19/Aug/19 22:36
Start Date: 19/Aug/19 22:36
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1263:
HDDS-1927. Consolidate add/remove Acl into OzoneAclUtil class. Contri…
URL: https://github.com/apache/hadoop/pull/1263#discussion_r315440404
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java
##
@@ -235,123 +231,22 @@ public FileEncryptionInfo getFileEncryptionInfo() {
return encInfo;
}
- public List getAcls() {
+ public List getAcls() {
Review comment:
We can leave this as OzoneAclInfo, and make OmBucketInfo OzoneAcl ->
OzoneAclInfo.
In this way, we can avoid conversion of OzoneAcl list during OM operations
(like add/remove/set Acl we can avoid protobuf conversion).
And this will be useful in HA code path, as we operate directly OMRequest
objects. (I mean here we get the data from OMRequest, and perform operation and
store in cache)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297479)
Time Spent: 2.5h (was: 2h 20m)
> Consolidate add/remove Acl into OzoneAclUtil class
> --
>
> Key: HDDS-1927
> URL: https://issues.apache.org/jira/browse/HDDS-1927
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> This Jira is created based on @xiaoyu comment on HDDS-1884
> Can we abstract these add/remove logic into common AclUtil class as we can
> see similar logic in both bucket manager and key manager? For example,
> public static boolean addAcl(List existingAcls, OzoneAcl newAcl)
> public static boolean removeAcl(List existingAcls, OzoneAcl newAcl)
>
> But to do this, we need both OmKeyInfo and OMBucketInfo to use list of
> OzoneAcl/OzoneAclInfo.
> This Jira is to do that refactor, and also address above comment to move
> common logic to AclUtils.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
