[jira] [Commented] (HDDS-1812) Du while calculating used disk space reports that chunk files are file not found
[
https://issues.apache.org/jira/browse/HDDS-1812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978817#comment-16978817
]
Anu Engineer commented on HDDS-1812:
bq. Do we really need the info on space used by Datanode? It does not seem
suitable for decisions regarding allocation, since the disk may be full with
other data.
I am ok with removing this information. As you mentioned it might not be very
useful for SCM to know.
> Du while calculating used disk space reports that chunk files are file not
> found
>
>
> Key: HDDS-1812
> URL: https://issues.apache.org/jira/browse/HDDS-1812
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Attila Doroszlai
>Priority: Critical
>
> {code}
> 2019-07-16 08:16:49,787 WARN org.apache.hadoop.fs.CachingGetSpaceUsed: Could
> not get disk usage information for path /data/3/ozone-0715
> ExitCodeException exitCode=1: du: cannot access
> '/data/3/ozone-0715/hdds/1b467d25-46cd-4de0-a4a1-e9405bde23ff/current/containerDir3/1724/chunks/b113dd390e68e914d3ff405f3deec564_stream_60448f
> 77-6349-48fa-ae86-b2d311730569_chunk_1.tmp.1.14118085': No such file or
> directory
> du: cannot access
> '/data/3/ozone-0715/hdds/1b467d25-46cd-4de0-a4a1-e9405bde23ff/current/containerDir3/1724/chunks/37993af2849bdd0320d0f9d4a6ef4b92_stream_1f68be9f-e083-45e5-84a9-08809bc392ed
> _chunk_1.tmp.1.14118091': No such file or directory
> du: cannot access
> '/data/3/ozone-0715/hdds/1b467d25-46cd-4de0-a4a1-e9405bde23ff/current/containerDir3/1724/chunks/a38677def61389ec0be9105b1b4fddff_stream_9c3c3741-f710-4482-8423-7ac6695be96b
> _chunk_1.tmp.1.14118102': No such file or directory
> du: cannot access
> '/data/3/ozone-0715/hdds/1b467d25-46cd-4de0-a4a1-e9405bde23ff/current/containerDir3/1724/chunks/a689c89f71a75547471baf6182f3be01_stream_baf0f21d-2fb0-4cd8-84b0-eff1723019a0
> _chunk_1.tmp.1.14118105': No such file or directory
> du: cannot access
> '/data/3/ozone-0715/hdds/1b467d25-46cd-4de0-a4a1-e9405bde23ff/current/containerDir3/1724/chunks/f58cf0fa5cb9360058ae25e8bc983e84_stream_d8d5ea61-995f-4ff5-88fb-4a9e97932f00
> _chunk_1.tmp.1.14118109': No such file or directory
> du: cannot access
> '/data/3/ozone-0715/hdds/1b467d25-46cd-4de0-a4a1-e9405bde23ff/current/containerDir3/1724/chunks/a1d13ee6bbefd1f8156b1bd8db0d1b67_stream_db214bdd-a0c0-4f4a-8bc7-a3817e047e45_chunk_1.tmp.1.14118115':
> No such file or directory
> du: cannot access
> '/data/3/ozone-0715/hdds/1b467d25-46cd-4de0-a4a1-e9405bde23ff/current/containerDir3/1724/chunks/8f8a4bd3f6c31161a70f82cb5ab8ee60_stream_d532d657-3d87-4332-baf8-effad9b3db23_chunk_1.tmp.1.14118127':
> No such file or directory
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008)
> at org.apache.hadoop.util.Shell.run(Shell.java:901)
> at org.apache.hadoop.fs.DU$DUShell.startRefresh(DU.java:62)
> at org.apache.hadoop.fs.DU.refresh(DU.java:53)
> at
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:181)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2512) Sonar TraceAllMethod NPE Could be Thrown
[ https://issues.apache.org/jira/browse/HDDS-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2512. Fix Version/s: 0.5.0 Resolution: Fixed [~MatthewSharp] Thanks for the contribution. [~adoroszlai] Thanks for the reviews. I have committed this patch to the master branch. > Sonar TraceAllMethod NPE Could be Thrown > > > Key: HDDS-2512 > URL: https://issues.apache.org/jira/browse/HDDS-2512 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Matthew Sharp >Assignee: Matthew Sharp >Priority: Minor > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Sonar cleanup: > [https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-2WKcVY8lQ4ZsNQ=AW5md-2WKcVY8lQ4ZsNQ] > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2541) CI builds should use merged code state instead of the forked branch
[ https://issues.apache.org/jira/browse/HDDS-2541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2541. Fix Version/s: 0.5.0 Resolution: Won't Fix Closing based on comments in the PR. Please re-open if needed. > CI builds should use merged code state instead of the forked branch > --- > > Key: HDDS-2541 > URL: https://issues.apache.org/jira/browse/HDDS-2541 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 10m > Remaining Estimate: 0h > > As of now the github actions based CI runs uses the branch of the PR which is > the forked repo most of the time. > It would be better to force a rebase/merge (without push) before the builds > to test the possible state after the merge not before. > For example if a PR branch uses elek/hadoop-ozone:HDDS-1234 and request a > merge to apache/hadoop-ozone:master then the build should download the > HDDS-1234 from elek/hadoop-ozone AND *rebase/merge* to the > apache/hadoop-ozone *before* the build. > This merge is temporary just for the build/checks (no push at all). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2543) Format specifiers should be used instead of string concatenation
[ https://issues.apache.org/jira/browse/HDDS-2543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2543. Fix Version/s: 0.5.0 Resolution: Fixed committed to the master branch. > Format specifiers should be used instead of string concatenation > > > Key: HDDS-2543 > URL: https://issues.apache.org/jira/browse/HDDS-2543 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Abhishek Purohit >Assignee: Abhishek Purohit >Priority: Minor > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Related to : > [https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md_AGKcVY8lQ4ZsV-=AW5md_AGKcVY8lQ4ZsV-] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2544) No need to call "toString()" method as formatting and string conversion is done by the Formatter.
[ https://issues.apache.org/jira/browse/HDDS-2544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2544. Fix Version/s: 0.5.0 Resolution: Fixed Thank you for the contribution. I have committed this patch to the master branch. > No need to call "toString()" method as formatting and string conversion is > done by the Formatter. > - > > Key: HDDS-2544 > URL: https://issues.apache.org/jira/browse/HDDS-2544 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Abhishek Purohit >Assignee: Abhishek Purohit >Priority: Minor > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Related to: > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md_AGKcVY8lQ4ZsWC=AW5md_AGKcVY8lQ4ZsWC -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2442) Add ServiceName support for getting Signed Cert.
[ https://issues.apache.org/jira/browse/HDDS-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2442. Fix Version/s: 0.5.0 Resolution: Fixed [~apurohit] Thank you for the contribution. I have committed this patch to the master. > Add ServiceName support for getting Signed Cert. > > > Key: HDDS-2442 > URL: https://issues.apache.org/jira/browse/HDDS-2442 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Anu Engineer >Assignee: Abhishek Purohit >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > We need to add support for adding Service name into the Certificate Signing > Request. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2546) Reorder the modifiers to comply with the Java Language Specification
[ https://issues.apache.org/jira/browse/HDDS-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2546. Fix Version/s: 0.5.0 Resolution: Fixed Thank you for the contribution. I have committed this patch to the master. > Reorder the modifiers to comply with the Java Language Specification > > > Key: HDDS-2546 > URL: https://issues.apache.org/jira/browse/HDDS-2546 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Abhishek Purohit >Assignee: Abhishek Purohit >Priority: Minor > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Related to : > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md_AbKcVY8lQ4ZsWo=AW5md_AbKcVY8lQ4ZsWo -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2548) The return type of this method should be an interface such as "ConcurrentMap" rather than the implementation "ConcurrentHashMap"
[ https://issues.apache.org/jira/browse/HDDS-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2548. Fix Version/s: 0.5.0 Resolution: Fixed Thank you for the contribution. I have committed this to the master branch. > The return type of this method should be an interface such as "ConcurrentMap" > rather than the implementation "ConcurrentHashMap" > > > Key: HDDS-2548 > URL: https://issues.apache.org/jira/browse/HDDS-2548 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Abhishek Purohit >Assignee: Abhishek Purohit >Priority: Minor > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Related to : > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md_AKKcVY8lQ4ZsWH=AW5md_AKKcVY8lQ4ZsWH -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2547) Sonar: remove volatile keyword from BlockOutputStream blockID field (#79)
[ https://issues.apache.org/jira/browse/HDDS-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2547. Resolution: Fixed [~MohammadJKhan] Thanks for the contribution. I have committed this patch to the master branch. > Sonar: remove volatile keyword from BlockOutputStream blockID field (#79) > - > > Key: HDDS-2547 > URL: https://issues.apache.org/jira/browse/HDDS-2547 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Mohammad >Assignee: Mohammad >Priority: Minor > Labels: pull-request-available, pull-requests-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Sonar report : > [https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-_2KcVY8lQ4ZsVd=false=BUG|https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-4jKcVY8lQ4ZsPQ=AW5md-4jKcVY8lQ4ZsPQ] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2372) Datanode pipeline is failing with NoSuchFileException
[
https://issues.apache.org/jira/browse/HDDS-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974483#comment-16974483
]
Anu Engineer commented on HDDS-2372:
bq. It's possible to remove the usage of the tmp files but only if we allow
overwrite for all the chunk files (in case of a leader failure the next attempt
to write may find the previous chunk file in place). It may be accepted but
it's a change with more risk.
Why this is an enforced constraint? It is the artifact of our code. It should
be trivial to check if file exists , and write chunk_file_v1, chunk_file_v2
etc. Anyway, as you mentioned, we will anyway rewrite this whole path. So it is
probably ok to do what you think works now.
> Datanode pipeline is failing with NoSuchFileException
> -
>
> Key: HDDS-2372
> URL: https://issues.apache.org/jira/browse/HDDS-2372
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Marton Elek
>Assignee: Marton Elek
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Found it on a k8s based test cluster using a simple 3 node cluster and
> HDDS-2327 freon test. After a while the StateMachine become unhealthy after
> this error:
> {code:java}
> datanode-0 datanode java.util.concurrent.ExecutionException:
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> java.nio.file.NoSuchFileException:
> /data/storage/hdds/2a77fab9-9dc5-4f73-9501-b5347ac6145c/current/containerDir0/1/chunks/gGYYgiTTeg_testdata_chunk_13931.tmp.2.20830
> {code}
> Can be reproduced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2481) Close streams in TarContainerPacker
[
https://issues.apache.org/jira/browse/HDDS-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDDS-2481:
---
Fix Version/s: 0.5.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thank you for fixing this issue. I greatly appreciate it. I have committed this
patch to the master.
> Close streams in TarContainerPacker
> ---
>
> Key: HDDS-2481
> URL: https://issues.apache.org/jira/browse/HDDS-2481
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Minor
> Labels: pull-request-available, sonar
> Fix For: 0.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Ensure various streams are closed in {{TarContainerPacker}}:
> *
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-9bKcVY8lQ4ZsUH=AW5md-9bKcVY8lQ4ZsUH
> *
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-9bKcVY8lQ4ZsUL=AW5md-9bKcVY8lQ4ZsUL
> *
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-9bKcVY8lQ4ZsUK=AW5md-9bKcVY8lQ4ZsUK
> *
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-9bKcVY8lQ4ZsUJ=AW5md-9bKcVY8lQ4ZsUJ
> *
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-9bKcVY8lQ4ZsUI=AW5md-9bKcVY8lQ4ZsUI
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2473) Fix code reliability issues found by Sonar in Ozone Recon module.
[ https://issues.apache.org/jira/browse/HDDS-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2473. Resolution: Fixed Thank you for an excellent patch. Much appreciated. I have committed this to the Master. > Fix code reliability issues found by Sonar in Ozone Recon module. > - > > Key: HDDS-2473 > URL: https://issues.apache.org/jira/browse/HDDS-2473 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Recon >Affects Versions: 0.5.0 >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > sonarcloud.io has flagged a number of code reliability issues in Ozone recon > (https://sonarcloud.io/code?id=hadoop-ozone=hadoop-ozone%3Ahadoop-ozone%2Frecon%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fozone%2Frecon). > Following issues will be triaged / fixed. > * Double Brace Initialization should not be used > * Resources should be closed > * InterruptedException should not be ignored -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2450) Datanode ReplicateContainer thread pool should be configurable
[ https://issues.apache.org/jira/browse/HDDS-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-2450: --- Fix Version/s: 0.5.0 Resolution: Fixed Status: Resolved (was: Patch Available) [~avijayan] Thanks for the review. [~sodonnell] Thanks for the contribution. I have committed this patch to the Master branch. > Datanode ReplicateContainer thread pool should be configurable > -- > > Key: HDDS-2450 > URL: https://issues.apache.org/jira/browse/HDDS-2450 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Affects Versions: 0.5.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The replicateContainer command uses a ReplicationSupervisor object to > implement a threadpool used to process replication commands. > In DatanodeStateMachine this thread pool is initialized with a hard coded > number of threads (10). This should be made configurable with a default value > of 10. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2479) Sonar : replace instanceof with catch block in XceiverClientGrpc.sendCommandWithRetry
[ https://issues.apache.org/jira/browse/HDDS-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2479. Fix Version/s: 0.5.0 Resolution: Fixed > Sonar : replace instanceof with catch block in > XceiverClientGrpc.sendCommandWithRetry > - > > Key: HDDS-2479 > URL: https://issues.apache.org/jira/browse/HDDS-2479 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Minor > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Sonar issue: > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md_AGKcVY8lQ4ZsV_=AW5md_AGKcVY8lQ4ZsV_ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2479) Sonar : replace instanceof with catch block in XceiverClientGrpc.sendCommandWithRetry
[ https://issues.apache.org/jira/browse/HDDS-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974437#comment-16974437 ] Anu Engineer commented on HDDS-2479: Thank you for the contribution. I have committed this patch to the master. > Sonar : replace instanceof with catch block in > XceiverClientGrpc.sendCommandWithRetry > - > > Key: HDDS-2479 > URL: https://issues.apache.org/jira/browse/HDDS-2479 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Minor > Labels: pull-request-available, sonar > Time Spent: 20m > Remaining Estimate: 0h > > Sonar issue: > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md_AGKcVY8lQ4ZsV_=AW5md_AGKcVY8lQ4ZsV_ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2480) Sonar : remove log spam for exceptions inside XceiverClientGrpc.reconnect
[ https://issues.apache.org/jira/browse/HDDS-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2480. Fix Version/s: 0.5.0 Resolution: Fixed [~sdeka] Thank you for the contribution. I have committed this patch to the master branch. > Sonar : remove log spam for exceptions inside XceiverClientGrpc.reconnect > - > > Key: HDDS-2480 > URL: https://issues.apache.org/jira/browse/HDDS-2480 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Minor > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Sonar issue: > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md_AGKcVY8lQ4ZsWE=AW5md_AGKcVY8lQ4ZsWE -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2478) Sonar : remove temporary variable in XceiverClientGrpc.sendCommand
[ https://issues.apache.org/jira/browse/HDDS-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-2478: --- Fix Version/s: 0.5.0 Resolution: Fixed Status: Resolved (was: Patch Available) [~adoroszlai] Thanks for the review. [~sdeka] Thanks for the patch. > Sonar : remove temporary variable in XceiverClientGrpc.sendCommand > -- > > Key: HDDS-2478 > URL: https://issues.apache.org/jira/browse/HDDS-2478 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Minor > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Sonar issues : > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md_AGKcVY8lQ4ZsV1=AW5md_AGKcVY8lQ4ZsV1 > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md_AGKcVY8lQ4ZsV2=AW5md_AGKcVY8lQ4ZsV2 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-2478) Sonar : remove temporary variable in XceiverClientGrpc.sendCommand
[ https://issues.apache.org/jira/browse/HDDS-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974428#comment-16974428 ] Anu Engineer edited comment on HDDS-2478 at 11/14/19 4:51 PM: -- [~adoroszlai] Thanks for the review. [~sdeka] Thanks for the patch. I have committed this to the master branch. was (Author: anu): [~adoroszlai] Thanks for the review. [~sdeka] Thanks for the patch. > Sonar : remove temporary variable in XceiverClientGrpc.sendCommand > -- > > Key: HDDS-2478 > URL: https://issues.apache.org/jira/browse/HDDS-2478 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Minor > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Sonar issues : > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md_AGKcVY8lQ4ZsV1=AW5md_AGKcVY8lQ4ZsV1 > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md_AGKcVY8lQ4ZsV2=AW5md_AGKcVY8lQ4ZsV2 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1940) Closing open container via scmcli gives false error message
[
https://issues.apache.org/jira/browse/HDDS-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDDS-1940:
---
Fix Version/s: 0.5.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
[~adoroszlai] Thanks for review and testing the patch. [~nanda] Thanks for the
contribution. I have committed this patch to the master branch.
> Closing open container via scmcli gives false error message
> ---
>
> Key: HDDS-1940
> URL: https://issues.apache.org/jira/browse/HDDS-1940
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Attila Doroszlai
>Assignee: Nanda kumar
>Priority: Minor
> Labels: incompatibleChange, pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{scmcli close}} prints an error message about invalid state transition after
> it had successfully closed the container.
> {code:title=CLI}
> $ ozone scmcli info 2
> ...
> Container State: OPEN
> ...
> $ ozone scmcli close 2
> ...
> client-09830A377AA9->f27bf787-8711-41d4-b0fd-3ef50b5c076f: receive
> RaftClientReply:client-09830A377AA9->f27bf787-8711-41d4-b0fd-3ef50b5c076f@group-7831D6F2EF1B,
> cid=0, SUCCESS, logIndex=11,
> commits[f27bf787-8711-41d4-b0fd-3ef50b5c076f:c12,
> 37ba33fe-c9ed-4ac2-a6e5-57ce658168b4:c11,
> feb68ba4-0a8a-4eda-9915-7dc090e5f46c:c11]
> Failed to update container state #2, reason: invalid state transition from
> state: CLOSED upon event: CLOSE.
> $ ozone scmcli info 2
> ...
> Container State: CLOSED
> ...
> {code}
> {code:title=logs}
> scm_1 | 2019-08-09 15:15:01 [IPC Server handler 1 on 9860] INFO
> SCMClientProtocolServer:366 - Object type container id 1 op close new stage
> begin
> dn3_1 | 2019-08-09 15:15:02 [RatisApplyTransactionExecutor 1] INFO
> Container:356 - Container 1 is closed with bcsId 3.
> dn1_1 | 2019-08-09 15:15:02 [RatisApplyTransactionExecutor 1] INFO
> Container:356 - Container 1 is closed with bcsId 3.
> scm_1 | 2019-08-09 15:15:02
> [EventQueue-IncrementalContainerReportForIncrementalContainerReportHandler]
> INFO IncrementalContainerReportHandler:176 - Moving container #1 to CLOSED
> state, datanode feb68ba4-0a8a-4eda-9915-7dc090e5f46c{ip: 10.5.1.6, host:
> ozone-static_dn3_1.ozone-static_net, networkLocation: /default-rack,
> certSerialId: null} reported CLOSED replica.
> dn2_1 | 2019-08-09 15:15:02 [RatisApplyTransactionExecutor 1] INFO
> Container:356 - Container 1 is closed with bcsId 3.
> scm_1 | 2019-08-09 15:15:02 [IPC Server handler 3 on 9860] INFO
> SCMClientProtocolServer:366 - Object type container id 1 op close new stage
> complete
> scm_1 | 2019-08-09 15:15:02 [IPC Server handler 3 on 9860] ERROR
> ContainerStateManager:335 - Failed to update container state #1, reason:
> invalid state transition from state: CLOSED upon event: CLOSE.
> scm_1 | 2019-08-09 15:15:02 [IPC Server handler 3 on 9860] INFO Server:2726
> - IPC Server handler 3 on 9860, call Call#3 Retry#0
> org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol.notifyObjectStageChange
> from 10.5.0.71:57746
> scm_1 | org.apache.hadoop.hdds.scm.exceptions.SCMException: Failed to update
> container state #1, reason: invalid state transition from state: CLOSED upon
> event: CLOSE.
> scm_1 | at
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.updateContainerState(ContainerStateManager.java:336)
> scm_1 | at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.updateContainerState(SCMContainerManager.java:312)
> scm_1 | at
> org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.notifyObjectStageChange(SCMClientProtocolServer.java:379)
> scm_1 | at
> org.apache.hadoop.ozone.protocolPB.StorageContainerLocationProtocolServerSideTranslatorPB.notifyObjectStageChange(StorageContainerLocationProtocolServerSideTranslatorPB.java:219)
> scm_1 | at
> org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java:16398)
> scm_1 | at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> scm_1 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> scm_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> scm_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> scm_1 | at java.base/java.security.AccessController.doPrivileged(Native
> Method)
> scm_1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> scm_1 | at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> scm_1 | at
[jira] [Resolved] (HDDS-2308) Switch to centos with the apache/ozone-build docker image
[
https://issues.apache.org/jira/browse/HDDS-2308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer resolved HDDS-2308.
Fix Version/s: 0.5.0
Resolution: Fixed
Committed to the build branch.
> Switch to centos with the apache/ozone-build docker image
> -
>
> Key: HDDS-2308
> URL: https://issues.apache.org/jira/browse/HDDS-2308
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Marton Elek
>Assignee: Marton Elek
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Attachments: hs_err_pid16346.log
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I realized multiple JVM crashes in the daily builds:
>
> {code:java}
> ERROR] ExecutionException The forked VM terminated without properly saying
> goodbye. VM crash or System.exit called?
>
>
> [ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs &&
> /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m
> -XX:+HeapDumpOnOutOfMemoryError -jar
> /workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter9018689154779946208.jar
> /workdir/hadoop-ozone/ozonefs/target/surefire
> 2019-10-06T14-52-40_697-jvmRun1 surefire7569723928289175829tmp
> surefire_947955725320624341206tmp
>
>
> [ERROR] Error occurred in starting fork, check output in log
>
>
> [ERROR] Process Exit Code: 139
>
>
> [ERROR] Crashed tests:
>
>
> [ERROR] org.apache.hadoop.fs.ozone.contract.ITestOzoneContractRename
>
>
> [ERROR] ExecutionException The forked VM terminated without properly
> saying goodbye. VM crash or System.exit called?
>
>
> [ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs &&
> /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m
> -XX:+HeapDumpOnOutOfMemoryError -jar
> /workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter5429192218879128313.jar
> /workdir/hadoop-ozone/ozonefs/target/surefire
> 2019-10-06T14-52-40_697-jvmRun1 surefire7227403571189445391tmp
> surefire_1011197392458143645283tmp
>
>
> [ERROR] Error occurred in starting fork, check output in log
>
>
> [ERROR] Process Exit Code: 139
>
>
> [ERROR] Crashed tests:
>
>
> [ERROR] org.apache.hadoop.fs.ozone.contract.ITestOzoneContractDistCp
>
>
> [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
>
>
> [ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs &&
> /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m
> -XX:+HeapDumpOnOutOfMemoryError -jar
> /workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter1355604543311368443.jar
> /workdir/hadoop-ozone/ozonefs/target/surefire
> 2019-10-06T14-52-40_697-jvmRun1 surefire3938612864214747736tmp
> surefire_933162535733309260236tmp
>
>
> [ERROR] Error occurred in starting fork, check output in log
>
>
> [ERROR] Process Exit Code: 139
>
>
> [ERROR] ExecutionException The forked VM terminated without properly
> saying goodbye. VM crash or System.exit called?
>
>
> [ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs &&
> /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m
> -XX:+HeapDumpOnOutOfMemoryError -jar
> /workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter9018689154779946208.jar
> /workdir/hadoop-ozone/ozonefs/target/surefire
> 2019-10-06T14-52-40_697-jvmRun1 surefire7569723928289175829tmp
> surefire_947955725320624341206tmp
>
>
> [ERROR] Error occurred in starting fork, check output in log
>
>
> [ERROR] Process Exit Code: 139 {code}
>
> Based on the crash log (uploaded) it's related to the rocksdb JNI interface.
> In the current ozone-build docker image (which provides the environment for
> build) we use alpine where musl libc is used instead of the main glibc. I
> think it would be more safe to use the same glibc what is used in production.
> I tested with centos based docker image and it seems to be more stable.
> Didn't see any more JVM crashes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1847) Datanode Kerberos principal and keytab config key looks inconsistent
[ https://issues.apache.org/jira/browse/HDDS-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-1847. Fix Version/s: 0.5.0 Resolution: Fixed [~chris.t...@gmail.com] Thanks for the contribution. [~elek] Thanks for retesting this patch. I have committed this change to the master branch. > Datanode Kerberos principal and keytab config key looks inconsistent > > > Key: HDDS-1847 > URL: https://issues.apache.org/jira/browse/HDDS-1847 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Eric Yang >Assignee: Chris Teoh >Priority: Major > Labels: newbie, pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Ozone Kerberos configuration can be very confusing: > | config name | Description | > | hdds.scm.kerberos.principal | SCM service principal | > | hdds.scm.kerberos.keytab.file | SCM service keytab file | > | ozone.om.kerberos.principal | Ozone Manager service principal | > | ozone.om.kerberos.keytab.file | Ozone Manager keytab file | > | hdds.scm.http.kerberos.principal | SCM service spnego principal | > | hdds.scm.http.kerberos.keytab.file | SCM service spnego keytab file | > | ozone.om.http.kerberos.principal | Ozone Manager spnego principal | > | ozone.om.http.kerberos.keytab.file | Ozone Manager spnego keytab file | > | hdds.datanode.http.kerberos.keytab | Datanode spnego keytab file | > | hdds.datanode.http.kerberos.principal | Datanode spnego principal | > | dfs.datanode.kerberos.principal | Datanode service principal | > | dfs.datanode.keytab.file | Datanode service keytab file | > The prefix are very different for each of the datanode configuration. It > would be nice to have some consistency for datanode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2364) Add a OM metrics to find the false positive rate for the keyMayExist
[ https://issues.apache.org/jira/browse/HDDS-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2364. Fix Version/s: 0.5.0 Resolution: Fixed [~avijayan] Thanks for the contribution. [~bharat] Thanks for the reviews. I have committed this to the master branch. > Add a OM metrics to find the false positive rate for the keyMayExist > > > Key: HDDS-2364 > URL: https://issues.apache.org/jira/browse/HDDS-2364 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.5.0 >Reporter: Mukul Kumar Singh >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Add a OM metrics to find the false positive rate for the keyMayExist. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2412) Define description/topics/merge strategy for the github repository with .asf.yaml
[ https://issues.apache.org/jira/browse/HDDS-2412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2412. Fix Version/s: 0.5.0 Resolution: Fixed Thanks, I have committed this patch to the master. [~elek] Thanks for the contribution. [~adoroszlai] Thanks for the reviews. > Define description/topics/merge strategy for the github repository with > .asf.yaml > - > > Key: HDDS-2412 > URL: https://issues.apache.org/jira/browse/HDDS-2412 > Project: Hadoop Distributed Data Store > Issue Type: Task >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > .asf.yaml helps to set different parameters on github repositories without > admin privileges: > [https://cwiki.apache.org/confluence/display/INFRA/.asf.yaml+features+for+git+repositories] > This basic .asf.yaml defines description/url/topics and the allowed merge > buttons. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2400) Enable github actions based builds for Ozone
[
https://issues.apache.org/jira/browse/HDDS-2400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer resolved HDDS-2400.
Fix Version/s: 0.5.0
Resolution: Fixed
Thanks, Committed to the master.
> Enable github actions based builds for Ozone
>
>
> Key: HDDS-2400
> URL: https://issues.apache.org/jira/browse/HDDS-2400
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: build
>Reporter: Marton Elek
>Assignee: Marton Elek
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Current PR checks are executed in a private branch based on the scripts in
> [https://github.com/elek/argo-ozone]
> but the results are stored in a public repositories:
> [https://github.com/elek/ozone-ci-q4|https://github.com/elek/ozone-ci-q3]
> [https://github.com/elek/ozone-ci-03]
>
> As we discussed during the community calls, it would be great to use github
> actions (or any other cloud based build) to make all the build definitions
> more accessible for the community.
> [~vivekratnavel] checked CircleCI which has better reporting capabilities.
> But INFRA has concerns about the permission model of circle-ci:
> {quote}it is highly unlikley we will allow a bot to be able to commit code
> (whether or not that is the intention, allowing circle-ci will make this
> possible, and is a complete no)
> {quote}
> See:
> https://issues.apache.org/jira/browse/INFRA-18131
> [https://lists.apache.org/thread.html/af52e2a3e865c01596d46374e8b294f2740587dbd59d85e132429b6c@%3Cbuilds.apache.org%3E]
>
> Fortunately we have a clear contract. Or build scripts are stored under
> _hadoop-ozone/dev-support/checks_ (return code show the result, details are
> printed out to the console output). It's very easy to experiment with
> different build systems.
>
> Github action seems to be an obvious choice: it's integrated well with GitHub
> and it has more generous resource limitations.
>
> With this Jira I propose to enable github actions based PR checks for a few
> tests (author, rat, unit, acceptance, checkstyle, findbugs) as an experiment.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2462) Add jq dependency in Contribution guideline
[ https://issues.apache.org/jira/browse/HDDS-2462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2462. Fix Version/s: 0.5.0 Resolution: Fixed Committed to Master branch. > Add jq dependency in Contribution guideline > --- > > Key: HDDS-2462 > URL: https://issues.apache.org/jira/browse/HDDS-2462 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Istvan Fajth >Assignee: Istvan Fajth >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Docker based tests are using JQ to parse JMX pages of different processes, > but the documentation does not mention it as a dependency. > Add it to CONTRIBUTION.MD in the "Additional requirements to execute > different type of tests" section. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2404) Add support for Registered id as service identifier for CSR.
[ https://issues.apache.org/jira/browse/HDDS-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2404. Fix Version/s: 0.5.0 Resolution: Fixed Committed to the master. > Add support for Registered id as service identifier for CSR. > > > Key: HDDS-2404 > URL: https://issues.apache.org/jira/browse/HDDS-2404 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Anu Engineer >Assignee: Abhishek Purohit >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The SCM HA needs the ability to represent a group as a single entity. So that > Tokens for each of the OM which is part of an HA group can be honored by the > data nodes. > This patch adds the notion of a service group ID to the Certificate > Infrastructure. In the next JIRAs, we will use this capability when issuing > certificates to OM -- especially when they are in HA mode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2442) Add ServiceName support for Certificate Signing Request.
Anu Engineer created HDDS-2442: -- Summary: Add ServiceName support for Certificate Signing Request. Key: HDDS-2442 URL: https://issues.apache.org/jira/browse/HDDS-2442 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: SCM Reporter: Anu Engineer Assignee: Abhishek Purohit We need to add support for adding Service name into the Certificate Signing Request. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2442) Add ServiceName support for Certificate Signing Request.
[ https://issues.apache.org/jira/browse/HDDS-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-2442: --- Parent: HDDS-505 Issue Type: Sub-task (was: Improvement) > Add ServiceName support for Certificate Signing Request. > > > Key: HDDS-2442 > URL: https://issues.apache.org/jira/browse/HDDS-2442 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Anu Engineer >Assignee: Abhishek Purohit >Priority: Major > > We need to add support for adding Service name into the Certificate Signing > Request. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2441) Add documentation for Empty-Trash command.
Anu Engineer created HDDS-2441: -- Summary: Add documentation for Empty-Trash command. Key: HDDS-2441 URL: https://issues.apache.org/jira/browse/HDDS-2441 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: documentation Reporter: Anu Engineer Add documentation for empty-trash command. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2440) Add empty-trash to ozone shell.
Anu Engineer created HDDS-2440: -- Summary: Add empty-trash to ozone shell. Key: HDDS-2440 URL: https://issues.apache.org/jira/browse/HDDS-2440 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone CLI Reporter: Anu Engineer Add emptry-trash command to Ozone shell. We should decide if we want to add this to the admin shell or normal shell. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2439) Add robot tests for empty-trash as owner.
Anu Engineer created HDDS-2439: -- Summary: Add robot tests for empty-trash as owner. Key: HDDS-2439 URL: https://issues.apache.org/jira/browse/HDDS-2439 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Anu Engineer We need to make sure that only Owner or Admins can execute the empty-trash command. We need to verify this using end-to-end tests, for example, robot tests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2438) Add the core logic for empty-trash
Anu Engineer created HDDS-2438: -- Summary: Add the core logic for empty-trash Key: HDDS-2438 URL: https://issues.apache.org/jira/browse/HDDS-2438 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Anu Engineer -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2437) Restrict empty-trash to admins and owners only
Anu Engineer created HDDS-2437: -- Summary: Restrict empty-trash to admins and owners only Key: HDDS-2437 URL: https://issues.apache.org/jira/browse/HDDS-2437 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Anu Engineer Make sure that only the owner of a key/adminstrator can empty-trash. The delete ACL is not enough to empty-trash. This is becasue a shared bucket can have deletes but the owner should be able to recover them. Once empty-trash is executed even the owner will be able to recover the deleted keys -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2436) Add security profile support for empty-trash command
Anu Engineer created HDDS-2436: -- Summary: Add security profile support for empty-trash command Key: HDDS-2436 URL: https://issues.apache.org/jira/browse/HDDS-2436 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Anu Engineer Add support for a certain groups to have the ability to have empty-trash. It might be the case where we want this command only to be run by admins. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2435) Add the ability to disable empty-trash command.
Anu Engineer created HDDS-2435: -- Summary: Add the ability to disable empty-trash command. Key: HDDS-2435 URL: https://issues.apache.org/jira/browse/HDDS-2435 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Anu Engineer Add a configuration key to disable empty-trash command. We can discuss if this should be a system-wide setting or per bucket. It is easier to do this system-wide I guess. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2434) Add server side support for empty-trash command.
Anu Engineer created HDDS-2434: -- Summary: Add server side support for empty-trash command. Key: HDDS-2434 URL: https://issues.apache.org/jira/browse/HDDS-2434 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Anu Engineer Add server side support for empty-trash command. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2433) Add client side support for the empty-trash command.
Anu Engineer created HDDS-2433: -- Summary: Add client side support for the empty-trash command. Key: HDDS-2433 URL: https://issues.apache.org/jira/browse/HDDS-2433 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Anu Engineer Add client side support for the empty-trash command. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2432) Add documentation for the recover-trash
Anu Engineer created HDDS-2432: -- Summary: Add documentation for the recover-trash Key: HDDS-2432 URL: https://issues.apache.org/jira/browse/HDDS-2432 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: documentation Reporter: Anu Engineer Add documentation for the recover-trash command in Ozone Documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2431) Add recover-trash command to the ozone shell.
Anu Engineer created HDDS-2431: -- Summary: Add recover-trash command to the ozone shell. Key: HDDS-2431 URL: https://issues.apache.org/jira/browse/HDDS-2431 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone CLI Reporter: Anu Engineer Add recover-trash command to the Ozone CLI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2430) Recover-trash should warn and skip if at-rest encryption is enabled and keys are missing.
Anu Engineer created HDDS-2430: -- Summary: Recover-trash should warn and skip if at-rest encryption is enabled and keys are missing. Key: HDDS-2430 URL: https://issues.apache.org/jira/browse/HDDS-2430 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Anu Engineer If TDE is enabled, recovering a key is useful only if the actual keys that are used for encryption are still recoverable. We should warn and fail the recovery if the actual keys are missing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2429) Recover-trash should warn and skip if the key is GDPR-ed key that recovery is pointless since the encryption keys are lost.
Anu Engineer created HDDS-2429: -- Summary: Recover-trash should warn and skip if the key is GDPR-ed key that recovery is pointless since the encryption keys are lost. Key: HDDS-2429 URL: https://issues.apache.org/jira/browse/HDDS-2429 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Anu Engineer If a bucket has GDPR enabled set, then it means that keys used to recover the data from the blocks is irrecoverably lost. In that case, a recover from trash is pointless. The recover-trash command should detect this case and let the users know about it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2428) Rename a recovered file as .recovered if the file already exists in the target bucket.
Anu Engineer created HDDS-2428: -- Summary: Rename a recovered file as .recovered if the file already exists in the target bucket. Key: HDDS-2428 URL: https://issues.apache.org/jira/browse/HDDS-2428 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Anu Engineer During recovery if the file name exists in the bucket, then the new key that is being recovered should be automatically renamed. The proposal is to rename it as key.recovered. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2426) Support recover-trash to an existing bucket.
Anu Engineer created HDDS-2426: -- Summary: Support recover-trash to an existing bucket. Key: HDDS-2426 URL: https://issues.apache.org/jira/browse/HDDS-2426 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Anu Engineer Support recovering trash to an existing bucket. We should also add a config key that prevents this mode, so admins can force the recovery to a new bucket always. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2425) Support the ability to recover-trash to a new bucket.
Anu Engineer created HDDS-2425: -- Summary: Support the ability to recover-trash to a new bucket. Key: HDDS-2425 URL: https://issues.apache.org/jira/browse/HDDS-2425 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Anu Engineer recover-trash can be run to recover to an existing bucket or to a new bucket. If the bucket does not exist, the recover-trash command should create that bucket automatically. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2424) Add the recover-trash command server side handling.
Anu Engineer created HDDS-2424: -- Summary: Add the recover-trash command server side handling. Key: HDDS-2424 URL: https://issues.apache.org/jira/browse/HDDS-2424 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Anu Engineer Add the standard server side code for command handling. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2423) Add the recover-trash command client side code
[ https://issues.apache.org/jira/browse/HDDS-2423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-2423: --- Description: Add protobuf, RpcClient and ClientSideTranslator code for the recover-trash command. (was: Add protobuf, RpcClient and ClientSideTranslator code for the Empty-trash command.) > Add the recover-trash command client side code > -- > > Key: HDDS-2423 > URL: https://issues.apache.org/jira/browse/HDDS-2423 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Anu Engineer >Priority: Major > > Add protobuf, RpcClient and ClientSideTranslator code for the recover-trash > command. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2423) Add the recover-trash command client side code
Anu Engineer created HDDS-2423: -- Summary: Add the recover-trash command client side code Key: HDDS-2423 URL: https://issues.apache.org/jira/browse/HDDS-2423 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Anu Engineer Add protobuf, RpcClient and ClientSideTranslator code for the Empty-trash command. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2422) Add robot tests for list-trash command.
Anu Engineer created HDDS-2422: -- Summary: Add robot tests for list-trash command. Key: HDDS-2422 URL: https://issues.apache.org/jira/browse/HDDS-2422 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: test Reporter: Anu Engineer Add robot tests for list-trash command and add those tests to integration.sh so these commands are run as part of CI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2421) Add documentation for list trash command.
Anu Engineer created HDDS-2421: -- Summary: Add documentation for list trash command. Key: HDDS-2421 URL: https://issues.apache.org/jira/browse/HDDS-2421 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: documentation Reporter: Anu Engineer Add documentation about the list-trash command. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2420) Add the Ozone shell support for list-trash command.
Anu Engineer created HDDS-2420: -- Summary: Add the Ozone shell support for list-trash command. Key: HDDS-2420 URL: https://issues.apache.org/jira/browse/HDDS-2420 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone CLI Reporter: Anu Engineer Add support for list-trash command in Ozone CLI. Please see the attached design doc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2418) Add the list trash command server side handling.
Anu Engineer created HDDS-2418: -- Summary: Add the list trash command server side handling. Key: HDDS-2418 URL: https://issues.apache.org/jira/browse/HDDS-2418 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Anu Engineer Add the standard code for any command handling in the server side. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2419) Add the core logic to process list trash command.
Anu Engineer created HDDS-2419: -- Summary: Add the core logic to process list trash command. Key: HDDS-2419 URL: https://issues.apache.org/jira/browse/HDDS-2419 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Anu Engineer Add the core logic of reading from the deleted table, and return the entries that match the user query. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2417) Add the list trash command to the client side
Anu Engineer created HDDS-2417: -- Summary: Add the list trash command to the client side Key: HDDS-2417 URL: https://issues.apache.org/jira/browse/HDDS-2417 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Anu Engineer Add the list-trash command to the protobuf files and to the client side translator. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2372) Datanode pipeline is failing with NoSuchFileException
[
https://issues.apache.org/jira/browse/HDDS-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969470#comment-16969470
]
Anu Engineer commented on HDDS-2372:
> Thanks Anu Engineer for the suggestion. Writing to the actual chunk file may
> lead to handling truncation log entries in Ratis inside Ozone which we don't
> need to handle right now as we always write to tmp chunk files
That is correct. That is one of the reasons why we did the tmp way. But that
time we did not have the Data Scrubber thread. Now we do that a data scrubber
thread, so it is trivial for the chunkfile to be detected as junk and cleaned
up by this thread.
> Datanode pipeline is failing with NoSuchFileException
> -
>
> Key: HDDS-2372
> URL: https://issues.apache.org/jira/browse/HDDS-2372
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Marton Elek
>Assignee: Shashikant Banerjee
>Priority: Critical
>
> Found it on a k8s based test cluster using a simple 3 node cluster and
> HDDS-2327 freon test. After a while the StateMachine become unhealthy after
> this error:
> {code:java}
> datanode-0 datanode java.util.concurrent.ExecutionException:
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> java.nio.file.NoSuchFileException:
> /data/storage/hdds/2a77fab9-9dc5-4f73-9501-b5347ac6145c/current/containerDir0/1/chunks/gGYYgiTTeg_testdata_chunk_13931.tmp.2.20830
> {code}
> Can be reproduced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2372) Datanode pipeline is failing with NoSuchFileException
[
https://issues.apache.org/jira/browse/HDDS-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968767#comment-16968767
]
Anu Engineer commented on HDDS-2372:
In the Chunk write path, we write chunks to a temp file and then rename them to
the file file.
However, until we commit a block, any chunk file is a temp file for real since
no one can see the chunk file name until we commit the ChunkInfo into the
RocksDB.
So if we remove the tmpChunkFile and always write to the real chunk file, this
race condition will go away.
> Datanode pipeline is failing with NoSuchFileException
> -
>
> Key: HDDS-2372
> URL: https://issues.apache.org/jira/browse/HDDS-2372
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Marton Elek
>Assignee: Shashikant Banerjee
>Priority: Critical
>
> Found it on a k8s based test cluster using a simple 3 node cluster and
> HDDS-2327 freon test. After a while the StateMachine become unhealthy after
> this error:
> {code:java}
> datanode-0 datanode java.util.concurrent.ExecutionException:
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> java.nio.file.NoSuchFileException:
> /data/storage/hdds/2a77fab9-9dc5-4f73-9501-b5347ac6145c/current/containerDir0/1/chunks/gGYYgiTTeg_testdata_chunk_13931.tmp.2.20830
> {code}
> Can be reproduced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2404) Add support for Registered id as service identifier for CSR.
[ https://issues.apache.org/jira/browse/HDDS-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968633#comment-16968633 ] Anu Engineer commented on HDDS-2404: Done, thanks > Add support for Registered id as service identifier for CSR. > > > Key: HDDS-2404 > URL: https://issues.apache.org/jira/browse/HDDS-2404 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Anu Engineer >Assignee: Abhishek Purohit >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The SCM HA needs the ability to represent a group as a single entity. So that > Tokens for each of the OM which is part of an HA group can be honored by the > data nodes. > This patch adds the notion of a service group ID to the Certificate > Infrastructure. In the next JIRAs, we will use this capability when issuing > certificates to OM -- especially when they are in HA mode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2404) Add support for Registered id as service identifier for CSR.
[ https://issues.apache.org/jira/browse/HDDS-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-2404: --- Parent: HDDS-505 Issue Type: Sub-task (was: Improvement) > Add support for Registered id as service identifier for CSR. > > > Key: HDDS-2404 > URL: https://issues.apache.org/jira/browse/HDDS-2404 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Anu Engineer >Assignee: Abhishek Purohit >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The SCM HA needs the ability to represent a group as a single entity. So that > Tokens for each of the OM which is part of an HA group can be honored by the > data nodes. > This patch adds the notion of a service group ID to the Certificate > Infrastructure. In the next JIRAs, we will use this capability when issuing > certificates to OM -- especially when they are in HA mode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2384) Large chunks during write can have memory pressure on DN with multiple clients
[
https://issues.apache.org/jira/browse/HDDS-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967931#comment-16967931
]
Anu Engineer commented on HDDS-2384:
Nope, we can encode the data in a set of small packets or read them as a
sequence of small packets. Let us say, 8KB/64KB buffer and we read and write
the data continually to the underlying disk.
> Large chunks during write can have memory pressure on DN with multiple clients
> --
>
> Key: HDDS-2384
> URL: https://issues.apache.org/jira/browse/HDDS-2384
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Anu Engineer
>Priority: Major
> Labels: performance
>
> During large file writes, it ends up writing {{16 MB}} chunks.
> https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java#L691
> In large clusters, 100s of clients may connect to DN. In such cases,
> depending on the incoming write workload mem load on DN can increase
> significantly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2404) Add support for Registered id as service identifier for CSR.
[ https://issues.apache.org/jira/browse/HDDS-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967051#comment-16967051 ] Anu Engineer commented on HDDS-2404: [~xyao], [~bharat], [~arp] This is the start of a set of JIRAs that enable security for the OM HA. We start with the Service Name attribute for the OMs. Please take a look at the pull request if you get a chance. Thanks > Add support for Registered id as service identifier for CSR. > > > Key: HDDS-2404 > URL: https://issues.apache.org/jira/browse/HDDS-2404 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Anu Engineer >Assignee: Abhishek Purohit >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The SCM HA needs the ability to represent a group as a single entity. So that > Tokens for each of the OM which is part of an HA group can be honored by the > data nodes. > This patch adds the notion of a service group ID to the Certificate > Infrastructure. In the next JIRAs, we will use this capability when issuing > certificates to OM -- especially when they are in HA mode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2404) Add support for Registered id as service identifier for CSR.
Anu Engineer created HDDS-2404: -- Summary: Add support for Registered id as service identifier for CSR. Key: HDDS-2404 URL: https://issues.apache.org/jira/browse/HDDS-2404 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: SCM Reporter: Anu Engineer The SCM HA needs the ability to represent a group as a single entity. So that Tokens for each of the OM which is part of an HA group can be honored by the data nodes. This patch adds the notion of a service group ID to the Certificate Infrastructure. In the next JIRAs, we will use this capability when issuing certificates to OM -- especially when they are in HA mode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1847) Datanode Kerberos principal and keytab config key looks inconsistent
[ https://issues.apache.org/jira/browse/HDDS-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965157#comment-16965157 ] Anu Engineer commented on HDDS-1847: It was my mistake. I saw this patch did not have the CI run. The patch was posted against hadoop repo instead of hadoop-ozone. Chris works out of Australia and is quite new to Ozone. So I am made a call that we will commit this and follow up with other patches that will fix the issue. I think he has already filed the follow up patches. Now that it is reverted, it might be better to recommit with the patch against hadoop-ozone. > Datanode Kerberos principal and keytab config key looks inconsistent > > > Key: HDDS-1847 > URL: https://issues.apache.org/jira/browse/HDDS-1847 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Eric Yang >Assignee: Chris Teoh >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Ozone Kerberos configuration can be very confusing: > | config name | Description | > | hdds.scm.kerberos.principal | SCM service principal | > | hdds.scm.kerberos.keytab.file | SCM service keytab file | > | ozone.om.kerberos.principal | Ozone Manager service principal | > | ozone.om.kerberos.keytab.file | Ozone Manager keytab file | > | hdds.scm.http.kerberos.principal | SCM service spnego principal | > | hdds.scm.http.kerberos.keytab.file | SCM service spnego keytab file | > | ozone.om.http.kerberos.principal | Ozone Manager spnego principal | > | ozone.om.http.kerberos.keytab.file | Ozone Manager spnego keytab file | > | hdds.datanode.http.kerberos.keytab | Datanode spnego keytab file | > | hdds.datanode.http.kerberos.principal | Datanode spnego principal | > | dfs.datanode.kerberos.principal | Datanode service principal | > | dfs.datanode.keytab.file | Datanode service keytab file | > The prefix are very different for each of the datanode configuration. It > would be nice to have some consistency for datanode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2396) OM rocksdb core dump during writing
[ https://issues.apache.org/jira/browse/HDDS-2396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964800#comment-16964800 ] Anu Engineer commented on HDDS-2396: [~bharat] Does this stack look similar to the issuse that was fixed with using try-with-resources where we had forgotten to clean up some resources in RocksDB ? > OM rocksdb core dump during writing > --- > > Key: HDDS-2396 > URL: https://issues.apache.org/jira/browse/HDDS-2396 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.4.1 >Reporter: Li Cheng >Priority: Major > Attachments: hs_err_pid9340.log > > > Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say > it's VM0. > I use goofys as a fuse and enable ozone S3 gateway to mount ozone to a path > on VM0, while reading data from VM0 local disk and write to mount path. The > dataset has various sizes of files from 0 byte to GB-level and it has a > number of ~50,000 files. > > There happens core dump in rocksdb while it's occasional. > > Stack: [0x7f5891a23000,0x7f5891b24000], sp=0x7f5891b21bb8, free > space=1018k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [libc.so.6+0x151d60] __memmove_ssse3_back+0x1ae0 > C [librocksdbjni3192271038586903156.so+0x358fec] > rocksdb::MemTableInserter::PutCFImpl(unsigned int, rocksdb::Slice const&, > rocksdb::Slice const&, rocksdb: > :ValueType)+0x51c > C [librocksdbjni3192271038586903156.so+0x359d17] > rocksdb::MemTableInserter::PutCF(unsigned int, rocksdb::Slice const&, > rocksdb::Slice const&)+0x17 > C [librocksdbjni3192271038586903156.so+0x3513bc] > rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler*) const+0x45c > C [librocksdbjni3192271038586903156.so+0x354df9] > rocksdb::WriteBatchInternal::InsertInto(rocksdb::WriteThread::WriteGroup&, > unsigned long, rocksdb::ColumnFamilyMemTables*, rocksdb::FlushScheduler*, > bool, unsigned long, rocksdb::DB*, bool, bool, bool)+0x1f9 > C [librocksdbjni3192271038586903156.so+0x29fd79] > rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, > rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, > bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x24b9 > C [librocksdbjni3192271038586903156.so+0x2a0431] > rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, > rocksdb::WriteBatch*)+0x21 > C [librocksdbjni3192271038586903156.so+0x1a064c] > Java_org_rocksdb_RocksDB_write0+0xcc > J 7899 org.rocksdb.RocksDB.write0(JJJ)V (0 bytes) @ 0x7f58f1872dbe > [0x7f58f1872d00+0xbe] > J 10093% C1 > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions()V > (400 bytes) @ 0x7f58f2308b0c [0x7f58f2307a40+0x10cc] > j > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer$$Lambda$29.run()V+4 > j java.lang.Thread.run()V+11 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1847) Datanode Kerberos principal and keytab config key looks inconsistent
[ https://issues.apache.org/jira/browse/HDDS-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964390#comment-16964390 ] Anu Engineer commented on HDDS-1847: interesting, [~chris.t...@gmail.com] can you please take a look when you get a chance.? > Datanode Kerberos principal and keytab config key looks inconsistent > > > Key: HDDS-1847 > URL: https://issues.apache.org/jira/browse/HDDS-1847 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.5.0 >Reporter: Eric Yang >Assignee: Chris Teoh >Priority: Major > Labels: newbie, pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Ozone Kerberos configuration can be very confusing: > | config name | Description | > | hdds.scm.kerberos.principal | SCM service principal | > | hdds.scm.kerberos.keytab.file | SCM service keytab file | > | ozone.om.kerberos.principal | Ozone Manager service principal | > | ozone.om.kerberos.keytab.file | Ozone Manager keytab file | > | hdds.scm.http.kerberos.principal | SCM service spnego principal | > | hdds.scm.http.kerberos.keytab.file | SCM service spnego keytab file | > | ozone.om.http.kerberos.principal | Ozone Manager spnego principal | > | ozone.om.http.kerberos.keytab.file | Ozone Manager spnego keytab file | > | hdds.datanode.http.kerberos.keytab | Datanode spnego keytab file | > | hdds.datanode.http.kerberos.principal | Datanode spnego principal | > | dfs.datanode.kerberos.principal | Datanode service principal | > | dfs.datanode.keytab.file | Datanode service keytab file | > The prefix are very different for each of the datanode configuration. It > would be nice to have some consistency for datanode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2384) Large chunks during write can have memory pressure on DN with multiple clients
[
https://issues.apache.org/jira/browse/HDDS-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964348#comment-16964348
]
Anu Engineer commented on HDDS-2384:
Thank you for flagging this issue. I think this is a hard to solve problem in
the current architecture. I would like to explore some possibilitues on how we
can solve this issue.
1. We add the support for a buffer pool inside data node. A buffer pool would a
large chunk of memory that data node will pin and internally treat it as a set
of buffers that can be used for I/O. When we read or write data, we will also
use this buffer pool. That way, we can limit the maximum committed memory that
we will end up using for data path.
2. In order to do that, we will now need the ability to read data not in 16 MB
chunks, but perhaps in smaller 8KB kind of size(assuming the page size is going
to be 8KB in the buffer pool).
3. The advantage of such an approach is that we will read data only as much
memory we have, but the network layer still might have to buffer this data.
4. This also allows us to push back against a client that is sending or trying
to read too much data from the data node at any given time.
Question: Do you think such a change would address this issue ? If you have
other suggestions, I would love to hear them. Once more, thank you for flagging
this issue.
> Large chunks during write can have memory pressure on DN with multiple clients
> --
>
> Key: HDDS-2384
> URL: https://issues.apache.org/jira/browse/HDDS-2384
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Anu Engineer
>Priority: Major
> Labels: performance
>
> During large file writes, it ends up writing {{16 MB}} chunks.
> https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java#L691
> In large clusters, 100s of clients may connect to DN. In such cases,
> depending on the incoming write workload mem load on DN can increase
> significantly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2384) Large chunks during write can have memory pressure on DN with multiple clients
[
https://issues.apache.org/jira/browse/HDDS-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer reassigned HDDS-2384:
--
Assignee: Anu Engineer
> Large chunks during write can have memory pressure on DN with multiple clients
> --
>
> Key: HDDS-2384
> URL: https://issues.apache.org/jira/browse/HDDS-2384
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Anu Engineer
>Priority: Major
> Labels: performance
>
> During large file writes, it ends up writing {{16 MB}} chunks.
> https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java#L691
> In large clusters, 100s of clients may connect to DN. In such cases,
> depending on the incoming write workload mem load on DN can increase
> significantly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1847) Datanode Kerberos principal and keytab config key looks inconsistent
[ https://issues.apache.org/jira/browse/HDDS-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-1847. Fix Version/s: 0.5.0 Resolution: Fixed I have committed this patch to the master branch. > Datanode Kerberos principal and keytab config key looks inconsistent > > > Key: HDDS-1847 > URL: https://issues.apache.org/jira/browse/HDDS-1847 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.5.0 >Reporter: Eric Yang >Assignee: Chris Teoh >Priority: Major > Labels: newbie, pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Ozone Kerberos configuration can be very confusing: > | config name | Description | > | hdds.scm.kerberos.principal | SCM service principal | > | hdds.scm.kerberos.keytab.file | SCM service keytab file | > | ozone.om.kerberos.principal | Ozone Manager service principal | > | ozone.om.kerberos.keytab.file | Ozone Manager keytab file | > | hdds.scm.http.kerberos.principal | SCM service spnego principal | > | hdds.scm.http.kerberos.keytab.file | SCM service spnego keytab file | > | ozone.om.http.kerberos.principal | Ozone Manager spnego principal | > | ozone.om.http.kerberos.keytab.file | Ozone Manager spnego keytab file | > | hdds.datanode.http.kerberos.keytab | Datanode spnego keytab file | > | hdds.datanode.http.kerberos.principal | Datanode spnego principal | > | dfs.datanode.kerberos.principal | Datanode service principal | > | dfs.datanode.keytab.file | Datanode service keytab file | > The prefix are very different for each of the datanode configuration. It > would be nice to have some consistency for datanode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2370) Remove classpath in RunningWithHDFS.md ozone-hdfs/docker-compose as dir 'ozoneplugin' is not exist anymore
[
https://issues.apache.org/jira/browse/HDDS-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964296#comment-16964296
]
Anu Engineer commented on HDDS-2370:
Yeap, I am fine with removing it. I don't think we are testing or running the
plugin inside HDFS any more.
> Remove classpath in RunningWithHDFS.md ozone-hdfs/docker-compose as dir
> 'ozoneplugin' is not exist anymore
> --
>
> Key: HDDS-2370
> URL: https://issues.apache.org/jira/browse/HDDS-2370
> Project: Hadoop Distributed Data Store
> Issue Type: Task
> Components: documentation
>Reporter: luhuachao
>Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-2370.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In RunningWithHDFS.md
> {code:java}
> export
> HADOOP_CLASSPATH=/opt/ozone/share/hadoop/ozoneplugin/hadoop-ozone-datanode-plugin.jar{code}
> ozone-hdfs/docker-compose.yaml
>
> {code:java}
> environment:
> HADOOP_CLASSPATH: /opt/ozone/share/hadoop/ozoneplugin/*.jar
> {code}
> when i run hddsdatanodeservice as pulgin in hdfs datanode, it comes out with
> the error below , there is no constructor without parameter.
>
>
> {code:java}
> 2019-10-21 21:38:56,391 ERROR datanode.DataNode
> (DataNode.java:startPlugins(972)) - Unable to load DataNode plugins.
> Specified list of plugins: org.apache.hadoop.ozone.HddsDatanodeService
> java.lang.RuntimeException: java.lang.NoSuchMethodException:
> org.apache.hadoop.ozone.HddsDatanodeService.()
> {code}
> what i doubt is that, ozone-0.5 not support running as a plugin in hdfs
> datanode now ? if so,
> why donnot we remove doc RunningWithHDFS.md ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2383) Closing open container via SCMCli throws exception
[
https://issues.apache.org/jira/browse/HDDS-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16963214#comment-16963214
]
Anu Engineer commented on HDDS-2383:
Looks like the SCMCLI was trying to close a container that SCM is also trying
to close. Would it be possible to close a good working container and see if you
see this error again. If so, it is a bug. Thanks
> Closing open container via SCMCli throws exception
> --
>
> Key: HDDS-2383
> URL: https://issues.apache.org/jira/browse/HDDS-2383
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: Rajesh Balamohan
>Priority: Major
>
> This was observed in apache master branch.
> Closing the container via {{SCMCli}} throws the following exception, though
> the container ends up getting closed eventually.
> {noformat}
> 2019-10-30 02:44:41,794 INFO
> org.apache.hadoop.hdds.scm.block.SCMBlockDeletingService: Block deletion
> txnID mismatch in datanode 79626ba3-1957-46e5-a8b0-32d7f47fb801 for
> containerID 6. Datanode delete txnID: 0, SCM txnID: 1004
> 2019-10-30 02:44:41,810 INFO
> org.apache.hadoop.hdds.scm.container.IncrementalContainerReportHandler:
> Moving container #4 to CLOSED state, datanode
> 8885d4ba-228a-4fd2-bf5a-831f01594c6c{ip: 10.17.234.37, host:
> vd1327.halxg.cloudera.com, networkLocation: /default-rack, certSerialId:
> null} reported CLOSED replica.
> 2019-10-30 02:44:41,826 INFO
> org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer: Object type
> container id 4 op close new stage complete
> 2019-10-30 02:44:41,826 ERROR
> org.apache.hadoop.hdds.scm.container.ContainerStateManager: Failed to update
> container state #4, reason: invalid state transition from state: CLOSED upon
> event: CLOSE.
> 2019-10-30 02:44:41,826 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 6 on 9860, call Call#3 Retry#0
> org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol.submitRequest
> from 10.17.234.32:45926
> org.apache.hadoop.hdds.scm.exceptions.SCMException: Failed to update
> container state #4, reason: invalid state transition from state: CLOSED upon
> event: CLOSE.
> at
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.updateContainerState(ContainerStateManager.java:338)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.updateContainerState(SCMContainerManager.java:326)
> at
> org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.notifyObjectStageChange(SCMClientProtocolServer.java:388)
> at
> org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.notifyObjectStageChange(StorageContainerLocationProtocolServerSideTranslatorPB.java:303)
> at
> org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.processRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:158)
> at
> org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB$$Lambda$152/2036820231.apply(Unknown
> Source)
> at
> org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72)
> at
> org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.submitRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:112)
> at
> org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java:30454)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2366) Remove ozone.enabled flag
[ https://issues.apache.org/jira/browse/HDDS-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16963204#comment-16963204 ] Anu Engineer commented on HDDS-2366: [~cxorm] I have updated the wiki page after committing this. Thank you for the reminder. We will scrub documenation and wiki for references to enabled. > Remove ozone.enabled flag > - > > Key: HDDS-2366 > URL: https://issues.apache.org/jira/browse/HDDS-2366 > Project: Hadoop Distributed Data Store > Issue Type: Task >Reporter: Bharat Viswanadham >Assignee: Siddharth Wagle >Priority: Major > Labels: newbie, pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Now when ozone is started the start-ozone.sh/stop-ozone.sh script check > whether this property is enabled or not to start ozone services. Now, this > property and this check can be removed. > > This was needed when ozone is part of Hadoop, and we don't want to start > ozone services by default. Now there is no such requirement. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2366) Remove ozone.enabled flag
[ https://issues.apache.org/jira/browse/HDDS-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2366. Fix Version/s: 0.5.0 Resolution: Fixed Committed to the master branch. [~swagle] Thank you for the contribution. > Remove ozone.enabled flag > - > > Key: HDDS-2366 > URL: https://issues.apache.org/jira/browse/HDDS-2366 > Project: Hadoop Distributed Data Store > Issue Type: Task >Reporter: Bharat Viswanadham >Assignee: Siddharth Wagle >Priority: Major > Labels: newbie, pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Now when ozone is started the start-ozone.sh/stop-ozone.sh script check > whether this property is enabled or not to start ozone services. Now, this > property and this check can be removed. > > This was needed when ozone is part of Hadoop, and we don't want to start > ozone services by default. Now there is no such requirement. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-426) Add field modificationTime for Volume and Bucket
[
https://issues.apache.org/jira/browse/HDDS-426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962671#comment-16962671
]
Anu Engineer commented on HDDS-426:
---
[~arp] Thanks for catching that, reopening since this JIRA is for modification
time. not creation time.
> Add field modificationTime for Volume and Bucket
>
>
> Key: HDDS-426
> URL: https://issues.apache.org/jira/browse/HDDS-426
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Manager
>Reporter: Dinesh Chitlangia
>Assignee: YiSheng Lien
>Priority: Major
> Labels: newbie
>
> There are update operations that can be performed for Volume, Bucket and Key.
> While Key records the modification time, Volume and & Bucket do not capture
> this.
>
> This Jira proposes to add the required field to Volume and Bucket in order to
> capture the modficationTime.
>
> Current Status:
> {noformat}
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoVolume /dummyvol
> 2018-09-10 17:16:12 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "owner" : {
> "name" : "bilbo"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "dummyvol",
> "createdOn" : "Mon, 10 Sep 2018 17:11:32 GMT",
> "createdBy" : "bilbo"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoBucket /dummyvol/mybuck
> 2018-09-10 17:15:25 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "volumeName" : "dummyvol",
> "bucketName" : "mybuck",
> "createdOn" : "Mon, 10 Sep 2018 17:12:09 GMT",
> "acls" : [ {
> "type" : "USER",
> "name" : "hadoop",
> "rights" : "READ_WRITE"
> }, {
> "type" : "GROUP",
> "name" : "users",
> "rights" : "READ_WRITE"
> }, {
> "type" : "USER",
> "name" : "spark",
> "rights" : "READ_WRITE"
> } ],
> "versioning" : "DISABLED",
> "storageType" : "DISK"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoKey /dummyvol/mybuck/myk1
> 2018-09-10 17:19:43 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "modifiedOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "size" : 0,
> "keyName" : "myk1",
> "keyLocations" : [ ]
> }{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDDS-426) Add field modificationTime for Volume and Bucket
[
https://issues.apache.org/jira/browse/HDDS-426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer reopened HDDS-426:
---
> Add field modificationTime for Volume and Bucket
>
>
> Key: HDDS-426
> URL: https://issues.apache.org/jira/browse/HDDS-426
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Manager
>Reporter: Dinesh Chitlangia
>Assignee: YiSheng Lien
>Priority: Major
> Labels: newbie
>
> There are update operations that can be performed for Volume, Bucket and Key.
> While Key records the modification time, Volume and & Bucket do not capture
> this.
>
> This Jira proposes to add the required field to Volume and Bucket in order to
> capture the modficationTime.
>
> Current Status:
> {noformat}
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoVolume /dummyvol
> 2018-09-10 17:16:12 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "owner" : {
> "name" : "bilbo"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "dummyvol",
> "createdOn" : "Mon, 10 Sep 2018 17:11:32 GMT",
> "createdBy" : "bilbo"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoBucket /dummyvol/mybuck
> 2018-09-10 17:15:25 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "volumeName" : "dummyvol",
> "bucketName" : "mybuck",
> "createdOn" : "Mon, 10 Sep 2018 17:12:09 GMT",
> "acls" : [ {
> "type" : "USER",
> "name" : "hadoop",
> "rights" : "READ_WRITE"
> }, {
> "type" : "GROUP",
> "name" : "users",
> "rights" : "READ_WRITE"
> }, {
> "type" : "USER",
> "name" : "spark",
> "rights" : "READ_WRITE"
> } ],
> "versioning" : "DISABLED",
> "storageType" : "DISK"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoKey /dummyvol/mybuck/myk1
> 2018-09-10 17:19:43 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "modifiedOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "size" : 0,
> "keyName" : "myk1",
> "keyLocations" : [ ]
> }{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-426) Add field modificationTime for Volume and Bucket
[
https://issues.apache.org/jira/browse/HDDS-426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer resolved HDDS-426.
---
Fix Version/s: 0.5.0
Resolution: Fixed
Looks like HDDS-1551 added Creation Time to bucketInfo and HDDS-1620 added
creationTime to VolumeInfo.
> Add field modificationTime for Volume and Bucket
>
>
> Key: HDDS-426
> URL: https://issues.apache.org/jira/browse/HDDS-426
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Manager
>Reporter: Dinesh Chitlangia
>Assignee: YiSheng Lien
>Priority: Major
> Labels: newbie
> Fix For: 0.5.0
>
>
> There are update operations that can be performed for Volume, Bucket and Key.
> While Key records the modification time, Volume and & Bucket do not capture
> this.
>
> This Jira proposes to add the required field to Volume and Bucket in order to
> capture the modficationTime.
>
> Current Status:
> {noformat}
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoVolume /dummyvol
> 2018-09-10 17:16:12 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "owner" : {
> "name" : "bilbo"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "dummyvol",
> "createdOn" : "Mon, 10 Sep 2018 17:11:32 GMT",
> "createdBy" : "bilbo"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoBucket /dummyvol/mybuck
> 2018-09-10 17:15:25 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "volumeName" : "dummyvol",
> "bucketName" : "mybuck",
> "createdOn" : "Mon, 10 Sep 2018 17:12:09 GMT",
> "acls" : [ {
> "type" : "USER",
> "name" : "hadoop",
> "rights" : "READ_WRITE"
> }, {
> "type" : "GROUP",
> "name" : "users",
> "rights" : "READ_WRITE"
> }, {
> "type" : "USER",
> "name" : "spark",
> "rights" : "READ_WRITE"
> } ],
> "versioning" : "DISABLED",
> "storageType" : "DISK"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoKey /dummyvol/mybuck/myk1
> 2018-09-10 17:19:43 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "modifiedOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "size" : 0,
> "keyName" : "myk1",
> "keyLocations" : [ ]
> }{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2378) Remove "OZONE" as string used in tests
[
https://issues.apache.org/jira/browse/HDDS-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962303#comment-16962303
]
Anu Engineer commented on HDDS-2378:
Not sure I understand the evil we are trying to prevent here. But if it
entertains you and you are willing to do this; please be my guest. I will
provide the review bandwidth :)
> Remove "OZONE" as string used in tests
> --
>
> Key: HDDS-2378
> URL: https://issues.apache.org/jira/browse/HDDS-2378
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> Based on a review I have done a quick check, and there are quite a few places
> where we have hardcoded "ozone" as String literal or a capital version of it
> into the code.
> Let's check then one by one, and where it is possible replace it with
> OzoneConsts.OZONE, or if the lower case version is not acceptable at all
> places, then create an other constant with the uppercase version and use that.
> This is the search, and the results:
> {code:bash}
> find . -name *.java | while read FILE; do NUM=`grep -c -i "\"OZONE\"" $FILE`;
> if [ $NUM -gt 0 ]; then echo $FILE; fi; done | sort | uniq
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDataYaml.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestBlockManagerImpl.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java
> ./hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ServerUtils.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/SCMContainerManagerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/block/TestBlockManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestCloseContainerEventHandler.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSCMContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestContainerPlacement.java
> ./hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/CreateSubcommand.java
> ./hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/OzoneVersionInfo.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/metrics/TestSCMContainerManagerMetrics.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerOperations.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerStateMachineIdempotency.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestS3Shell.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestAllocateContainer.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSmallFile.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestGetCommittedBlockLengthAndPutKey.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientMetrics.java
>
[jira] [Resolved] (HDDS-2374) Make Ozone Readme.txt point to the Ozone websites instead of Hadoop.
[ https://issues.apache.org/jira/browse/HDDS-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2374. Fix Version/s: 0.5.0 Resolution: Fixed merged to the master > Make Ozone Readme.txt point to the Ozone websites instead of Hadoop. > > > Key: HDDS-2374 > URL: https://issues.apache.org/jira/browse/HDDS-2374 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > See the title. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2374) Make Ozone Readme.txt point to the Ozone websites instead of Hadoop.
Anu Engineer created HDDS-2374: -- Summary: Make Ozone Readme.txt point to the Ozone websites instead of Hadoop. Key: HDDS-2374 URL: https://issues.apache.org/jira/browse/HDDS-2374 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Anu Engineer Assignee: Anu Engineer See the title. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2349) QueryNode does not respect null values for opState or state
[ https://issues.apache.org/jira/browse/HDDS-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-2349: --- Fix Version/s: 0.5.0 Resolution: Fixed Status: Resolved (was: Patch Available) Thank you for the contribution. I have committed this patch to the decom branch. > QueryNode does not respect null values for opState or state > --- > > Key: HDDS-2349 > URL: https://issues.apache.org/jira/browse/HDDS-2349 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Affects Versions: 0.5.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > In HDDS-2197, the queryNode API call was changed to allow operational state > (in_service, decommissioning etc) to be passed along with the node health > state. This changed allowed for a null state to indicate a wildcard, so > passing: > opState = null > healthState = HEALTHY > Allows one to find all the healthy nodes, irrespective of their opState. > However, for an enum protobuf field, if no value is specified, the first enum > in the set is returned as the default. This means that when a null is passed > for opState, only the IN_SERVICE nodes are returned. Similar for health state > - passing a null will return only HEALTHY nodes. > This PR will fix this issue so the null value acts as a wildcard as intended. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2287) Move ozone source code to apache/hadoop-ozone from apache/hadoop
[ https://issues.apache.org/jira/browse/HDDS-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16958148#comment-16958148 ] Anu Engineer commented on HDDS-2287: [~vinodkv], we can think of HDDS as similar to Hadoop Common, but with more separation and built in shading. So eventually when HDFS consumes it, we will consume HDDS via a Maven package. That is what we were thinking, this allows the dependencies of HDDS to be fully isolated from all down stream, which is a big issue in the world of Hadoop. > Move ozone source code to apache/hadoop-ozone from apache/hadoop > > > Key: HDDS-2287 > URL: https://issues.apache.org/jira/browse/HDDS-2287 > Project: Hadoop Distributed Data Store > Issue Type: Task >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > > *This issue is created to use the assigned number for any technical commits > to make it easy to follow the root reason of the commit...* > > As discussed and voted on the mailing lists, Apache Hadoop Ozone source code > will be removed from the hadoop trunk and stored in a separated repository. > > Original discussion is here: > [https://lists.apache.org/thread.html/ef01b7def94ba58f746875999e419e10645437423ab9af19b32821e7@%3Chdfs-dev.hadoop.apache.org%3E] > (It's started as a discussion but as everybody started to vote it's finished > with a call to a lazy consensus vote) > > Technical proposal is shared on the wiki: > [https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Ozone+source+tree+split] > > Discussed on the community meeting: > [https://cwiki.apache.org/confluence/display/HADOOP/2019-09-30+Meeting+notes] > > Which is shared on the mailing list to get more feedback: > [https://lists.apache.org/thread.html/ed608c708ea302675ae5e39636ed73613f47a93c2ddfbd3c9e24dbae@%3Chdfs-dev.hadoop.apache.org%3E] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2196) Add CLI Commands and Protobuf messages to trigger decom states
[ https://issues.apache.org/jira/browse/HDDS-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-2196: --- Fix Version/s: 0.5.0 Resolution: Fixed Status: Resolved (was: Patch Available) Committed to the decom branch. > Add CLI Commands and Protobuf messages to trigger decom states > -- > > Key: HDDS-2196 > URL: https://issues.apache.org/jira/browse/HDDS-2196 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM, SCM Client >Affects Versions: 0.5.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > To all nodes to be decommissioned, recommissioned and put into maintenance, > we need a few commands. > These will be added to the existing "scm cli". 3 commands are proposed: > Decommission: > ozone scmcli dnadmin decommission hosta hostb hostc:port ... > Put nodes into maintenance: > osone scmcli dnadmin maintenance hosta hostb hostc:port ... <-endHours> > Take nodes out of maintenance or halt decommission: > ozone scmcli dnadmin recommission hosta hostb hostc:port > These 3 commands will call 3 new protobuf messages and they will be part of > the "StorageContainerLocationProtocol": > * DecommissionNodesRequestProto > * RecommissionNodesRequestProto > * StartMaintenanceNodesRequestProto > In additional a new class NodeDecommissionManager will be introduced that > will receive these commands and carry out the decommission steps. > In this patch NodeDecommissionManager is only a skeleton implementation to > receive the commands as this patch is mainly focused on getting the CLI > commands and protobuf messages in place. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2328) Support large-scale listing
[
https://issues.apache.org/jira/browse/HDDS-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956216#comment-16956216
]
Anu Engineer commented on HDDS-2328:
Agree. We should probably do what S3AFileSystem has done.
> Support large-scale listing
>
>
> Key: HDDS-2328
> URL: https://issues.apache.org/jira/browse/HDDS-2328
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Rajesh Balamohan
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: performance
>
> Large-scale listing of directory contents takes a lot longer time and also
> has the potential to run into OOM. I have > 1 million entries in the same
> level and it took lot longer time with {{RemoteIterator}} (didn't complete as
> it was stuck in RDB::seek).
> S3A batches it with 5K listing per fetch IIRC. It would be good to have this
> feature in ozone as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2323) Mem allocation: Optimise AuditMessage::build()
[
https://issues.apache.org/jira/browse/HDDS-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954902#comment-16954902
]
Anu Engineer commented on HDDS-2323:
I verified the TestAuditPaser works well. Since this can potentially cause a
failure there. It did pass for me.
> Mem allocation: Optimise AuditMessage::build()
> --
>
> Key: HDDS-2323
> URL: https://issues.apache.org/jira/browse/HDDS-2323
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Rajesh Balamohan
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: performance
> Fix For: 0.5.0
>
> Attachments: HDDS-2323.01.patch, Screenshot 2019-10-18 at 8.24.52
> AM.png
>
>
> String format allocates/processes more than
> {color:#00}OzoneAclUtil.fromProtobuf in write benchmark.{color}
> {color:#00}Would be good to use + instead of format.{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2323) Mem allocation: Optimise AuditMessage::build()
[
https://issues.apache.org/jira/browse/HDDS-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDDS-2323:
---
Fix Version/s: 0.5.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for finding this issue, appreciate it. [~rajesh.balamohan].
[~dineshchitlangia] Thanks for review and approval. [~swagle] Thank you for the
contribution. I have committed this to the master branch.
> Mem allocation: Optimise AuditMessage::build()
> --
>
> Key: HDDS-2323
> URL: https://issues.apache.org/jira/browse/HDDS-2323
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Rajesh Balamohan
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: performance
> Fix For: 0.5.0
>
> Attachments: HDDS-2323.01.patch, Screenshot 2019-10-18 at 8.24.52
> AM.png
>
>
> String format allocates/processes more than
> {color:#00}OzoneAclUtil.fromProtobuf in write benchmark.{color}
> {color:#00}Would be good to use + instead of format.{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2323) Mem allocation: Optimise AuditMessage::build()
[
https://issues.apache.org/jira/browse/HDDS-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954833#comment-16954833
]
Anu Engineer commented on HDDS-2323:
The change is so trivial, that I will test this on my laptop and commit it. As
[~dineshchitlangia] it is better to have pull requests with ozone due to the
CI. [~dineshchitlangia] I hope you don't see any major issues in me testing on
my machine and committing it ?
> Mem allocation: Optimise AuditMessage::build()
> --
>
> Key: HDDS-2323
> URL: https://issues.apache.org/jira/browse/HDDS-2323
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Rajesh Balamohan
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: performance
> Attachments: HDDS-2323.01.patch, Screenshot 2019-10-18 at 8.24.52
> AM.png
>
>
> String format allocates/processes more than
> {color:#00}OzoneAclUtil.fromProtobuf in write benchmark.{color}
> {color:#00}Would be good to use + instead of format.{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-2323) Mem allocation: Optimise AuditMessage::build()
[
https://issues.apache.org/jira/browse/HDDS-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954833#comment-16954833
]
Anu Engineer edited comment on HDDS-2323 at 10/18/19 5:13 PM:
--
The change is so trivial, that I will test this on my laptop and commit it. As
[~dineshchitlangia] said it is better to have pull requests with ozone due to
the CI. [~dineshchitlangia] I hope you don't see any major issues in me testing
on my machine and committing it ?
was (Author: anu):
The change is so trivial, that I will test this on my laptop and commit it. As
[~dineshchitlangia] it is better to have pull requests with ozone due to the
CI. [~dineshchitlangia] I hope you don't see any major issues in me testing on
my machine and committing it ?
> Mem allocation: Optimise AuditMessage::build()
> --
>
> Key: HDDS-2323
> URL: https://issues.apache.org/jira/browse/HDDS-2323
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Rajesh Balamohan
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: performance
> Attachments: HDDS-2323.01.patch, Screenshot 2019-10-18 at 8.24.52
> AM.png
>
>
> String format allocates/processes more than
> {color:#00}OzoneAclUtil.fromProtobuf in write benchmark.{color}
> {color:#00}Would be good to use + instead of format.{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2327) Provide new Freon test to test Ratis pipeline with pure XceiverClientRatis
[ https://issues.apache.org/jira/browse/HDDS-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954806#comment-16954806 ] Anu Engineer commented on HDDS-2327: Ok, I get it, even though this is a Ratis test case and ideally it should be in ratis; the code has too much dependency on the Ozone layer. Disregard my earlier comment. Though in the long run Ratis should have the ability to do these tests. > Provide new Freon test to test Ratis pipeline with pure XceiverClientRatis > -- > > Key: HDDS-2327 > URL: https://issues.apache.org/jira/browse/HDDS-2327 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: freon >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > [~xyao] suggested during an offline talk to implement one additional Freon > test to test the ratis part only. > It can use XceiverClientManager which creates a pure XceiverClientRatis. The > client can be used to generate chunks as the datanode accepts any container > id / block id. > With this approach we can stress-test one selected ratis pipeline without > having full end2end overhead of the key creation (OM, SCM, etc.) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2327) Provide new Freon test to test Ratis pipeline with pure XceiverClientRatis
[ https://issues.apache.org/jira/browse/HDDS-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954804#comment-16954804 ] Anu Engineer commented on HDDS-2327: why is this part of freon ? Should this be a Ratis test or tool which any ratis user would want ? and ozone should test with this, but should it not be via the binaries that come from Ratis ? > Provide new Freon test to test Ratis pipeline with pure XceiverClientRatis > -- > > Key: HDDS-2327 > URL: https://issues.apache.org/jira/browse/HDDS-2327 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: freon >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > [~xyao] suggested during an offline talk to implement one additional Freon > test to test the ratis part only. > It can use XceiverClientManager which creates a pure XceiverClientRatis. The > client can be used to generate chunks as the datanode accepts any container > id / block id. > With this approach we can stress-test one selected ratis pipeline without > having full end2end overhead of the key creation (OM, SCM, etc.) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2328) Support large-scale listing
[
https://issues.apache.org/jira/browse/HDDS-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954802#comment-16954802
]
Anu Engineer commented on HDDS-2328:
The Listing API interface already does that. I will take a look at why we are
not paging ... Can you please provide me with repro steps and which version of
branch you tried with this ?
> Support large-scale listing
>
>
> Key: HDDS-2328
> URL: https://issues.apache.org/jira/browse/HDDS-2328
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Rajesh Balamohan
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: performance
>
> Large-scale listing of directory contents takes a lot longer time and also
> has the potential to run into OOM. I have > 1 million entries in the same
> level and it took lot longer time with {{RemoteIterator}} (didn't complete as
> it was stuck in RDB::seek).
> S3A batches it with 5K listing per fetch IIRC. It would be good to have this
> feature in ozone as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2331) Client OOME due to buffer retention
[
https://issues.apache.org/jira/browse/HDDS-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954792#comment-16954792
]
Anu Engineer commented on HDDS-2331:
[~adoroszlai] Would you like me to revert this commit for now?
> Client OOME due to buffer retention
> ---
>
> Key: HDDS-2331
> URL: https://issues.apache.org/jira/browse/HDDS-2331
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.5.0
>Reporter: Attila Doroszlai
>Priority: Critical
> Attachments: profiler.png
>
>
> Freon random key generator exhausts default heap after just few hundred 1MB
> keys. Heap dump on OOME reveals 150+ instances of
> {{ContainerCommandRequestMessage}}, each with 16MB {{byte[]}}.
> Steps to reproduce:
> # Start Ozone cluster with 1 datanode
> # Start Freon (5K keys of size 1MB)
> Result: OOME after a few hundred keys
> {noformat}
> $ cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone
> $ docker-compose up -d
> $ docker-compose exec scm bash
> $ export HADOOP_OPTS='-XX:+HeapDumpOnOutOfMemoryError'
> $ ozone freon rk --numOfThreads 1 --numOfVolumes 1 --numOfBuckets 1
> --replicationType RATIS --factor ONE --keySize 1048576 --numOfKeys 5120
> --bufferSize 65536
> ...
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid289.hprof ...
> Heap dump file created [1456141975 bytes in 7.760 secs]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2254) Fix flaky unit testTestContainerStateMachine#testRatisSnapshotRetention
[
https://issues.apache.org/jira/browse/HDDS-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer resolved HDDS-2254.
Fix Version/s: 0.5.0
Resolution: Fixed
Committed to the master branch.
> Fix flaky unit testTestContainerStateMachine#testRatisSnapshotRetention
> ---
>
> Key: HDDS-2254
> URL: https://issues.apache.org/jira/browse/HDDS-2254
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Affects Versions: 0.5.0
>Reporter: Siddharth Wagle
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Test always fails with assertion error:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.ozone.client.rpc.TestContainerStateMachine.testRatisSnapshotRetention(TestContainerStateMachine.java:188)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2321) Ozone Block Token verify should not apply to all datanode cmd
[ https://issues.apache.org/jira/browse/HDDS-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954062#comment-16954062 ] Anu Engineer commented on HDDS-2321: Since SCM has the root cert, it might be intresting if it send a token over, that way these commands are also verified. In the long run, or even the short run, these SCM commands to DNs will go away. > Ozone Block Token verify should not apply to all datanode cmd > - > > Key: HDDS-2321 > URL: https://issues.apache.org/jira/browse/HDDS-2321 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.4.1 >Reporter: Nilotpal Nandi >Assignee: Xiaoyu Yao >Priority: Major > > DN container protocol has cmd send from SCM or other DN, which do not bear OM > block token like OM client. We should restrict the OM Block token check only > for those issued from OM client. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2302) Manage common pom versions in one common place
[ https://issues.apache.org/jira/browse/HDDS-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2302. Fix Version/s: 0.5.0 Resolution: Fixed > Manage common pom versions in one common place > -- > > Key: HDDS-2302 > URL: https://issues.apache.org/jira/browse/HDDS-2302 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Some of the versions (eg. ozone.version, hdds.version, ratis.version) are > required for both ozone and hdds subprojects. As we have a common pom.xml it > can be safer to manage them in one common place at the root pom.xml instead > of managing them multiple times. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2289) Put testing information and a problem description to the github PR template
[ https://issues.apache.org/jira/browse/HDDS-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2289. Resolution: Fixed > Put testing information and a problem description to the github PR template > --- > > Key: HDDS-2289 > URL: https://issues.apache.org/jira/browse/HDDS-2289 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Anu Engineer >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > This is suggested by [~aengineer] during an offline discussion to add more > information to the github PR template based on the template of ambari (by > Vivek): > https://github.com/apache/ambari/commit/579cec8cf5bcfe1a1a0feacf055ed6569f674e6a -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2181) Ozone Manager should send correct ACL type in ACL requests to Authorizer
[ https://issues.apache.org/jira/browse/HDDS-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953198#comment-16953198 ] Anu Engineer commented on HDDS-2181: [~elek] Didn't you revert this ? > Ozone Manager should send correct ACL type in ACL requests to Authorizer > > > Key: HDDS-2181 > URL: https://issues.apache.org/jira/browse/HDDS-2181 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.4.1 >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 10h 50m > Remaining Estimate: 0h > > Currently, Ozone manager sends "WRITE" as ACLType for key create, key delete > and bucket create operation. Fix the acl type in all requests to the > authorizer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDDS-2283) Container Creation on datanodes take around 300ms due to rocksdb creation
[ https://issues.apache.org/jira/browse/HDDS-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-2283: --- Comment: was deleted (was: I really doubt if this is a bottleneck. I am not disagreeing with the measurement. I am just saying in the big picture of writing 5GB of data, optimizing for this 300 ms buys you nothing. if you are optimizing for micro-benchmarks, what we will end of building is a system that is optimized for these benchmarks; not something that is really needed. ) > Container Creation on datanodes take around 300ms due to rocksdb creation > - > > Key: HDDS-2283 > URL: https://issues.apache.org/jira/browse/HDDS-2283 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Major > Labels: pull-request-available > Attachments: HDDS-2283.00.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Container Creation on datanodes take around 300ms due to rocksdb creation. > Rocksdb creation is taking a considerable time and this needs to be optimized. > Creating a rocksdb per disk should be enough and each container can be table > inside the rocksdb. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2283) Container Creation on datanodes take around 300ms due to rocksdb creation
[ https://issues.apache.org/jira/browse/HDDS-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953181#comment-16953181 ] Anu Engineer commented on HDDS-2283: I am fine this this change.. but the follow up JIRA makes no sense to me, my earlier deleted comment was about that. > Container Creation on datanodes take around 300ms due to rocksdb creation > - > > Key: HDDS-2283 > URL: https://issues.apache.org/jira/browse/HDDS-2283 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Major > Labels: pull-request-available > Attachments: HDDS-2283.00.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Container Creation on datanodes take around 300ms due to rocksdb creation. > Rocksdb creation is taking a considerable time and this needs to be optimized. > Creating a rocksdb per disk should be enough and each container can be table > inside the rocksdb. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2283) Container Creation on datanodes take around 300ms due to rocksdb creation
[ https://issues.apache.org/jira/browse/HDDS-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953179#comment-16953179 ] Anu Engineer commented on HDDS-2283: I really doubt if this is a bottleneck. I am not disagreeing with the measurement. I am just saying in the big picture of writing 5GB of data, optimizing for this 300 ms buys you nothing. if you are optimizing for micro-benchmarks, what we will end of building is a system that is optimized for these benchmarks; not something that is really needed. > Container Creation on datanodes take around 300ms due to rocksdb creation > - > > Key: HDDS-2283 > URL: https://issues.apache.org/jira/browse/HDDS-2283 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Major > Labels: pull-request-available > Attachments: HDDS-2283.00.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Container Creation on datanodes take around 300ms due to rocksdb creation. > Rocksdb creation is taking a considerable time and this needs to be optimized. > Creating a rocksdb per disk should be enough and each container can be table > inside the rocksdb. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2317) Change rocksDB per Container model to have table per container on RocksDb per disk
[ https://issues.apache.org/jira/browse/HDDS-2317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953176#comment-16953176 ] Anu Engineer commented on HDDS-2317: and how do we propose to move a container from one machine to another now? > Change rocksDB per Container model to have table per container on RocksDb per > disk > -- > > Key: HDDS-2317 > URL: https://issues.apache.org/jira/browse/HDDS-2317 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Affects Versions: 0.5.0 >Reporter: Siddharth Wagle >Priority: Major > > Idea proposed by [~msingh] in HDDS-2283. > Better utilize disk bandwidth by having Rocks DB per disk and put containers > as tables inside. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2317) Change rocksDB per Container model to have table per container on RocksDb per disk
[ https://issues.apache.org/jira/browse/HDDS-2317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953175#comment-16953175 ] Anu Engineer commented on HDDS-2317: Don't we create a container once per 5GB of data? are we optimizing for the micro-bench-marking scenarios? I am really surprised to learn that 300ms for a DB creation is the bottleneck for 5GB data write. > Change rocksDB per Container model to have table per container on RocksDb per > disk > -- > > Key: HDDS-2317 > URL: https://issues.apache.org/jira/browse/HDDS-2317 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Affects Versions: 0.5.0 >Reporter: Siddharth Wagle >Priority: Major > > Idea proposed by [~msingh] in HDDS-2283. > Better utilize disk bandwidth by having Rocks DB per disk and put containers > as tables inside. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2316) Support to skip recon and/or ozonefs during the build
[ https://issues.apache.org/jira/browse/HDDS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-2316. Fix Version/s: 0.5.0 Resolution: Fixed Committed to the master. > Support to skip recon and/or ozonefs during the build > - > > Key: HDDS-2316 > URL: https://issues.apache.org/jira/browse/HDDS-2316 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Anu Engineer >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > (I almost use this Jira summary: "Fast-lane to ozone build" It was very hard > to resist...) > > The two slowest part of Ozone build as of now: > # The (multiple) shading of ozonefs > # And the frontend build/obfuscation of ozone recon > [~aengineer] suggested to introduce options to skip them as they are not > required for the build all the time. > This patch introduces '-DskipRecon' and '-DskipShade' options to provide a > faster way to create a *partial* build. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2316) Support to skip recon and/or ozonefs during the build
[ https://issues.apache.org/jira/browse/HDDS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952849#comment-16952849 ] Anu Engineer commented on HDDS-2316: Thank you for the fast-lane. I am taking the first ride. > Support to skip recon and/or ozonefs during the build > - > > Key: HDDS-2316 > URL: https://issues.apache.org/jira/browse/HDDS-2316 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Anu Engineer >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > (I almost use this Jira summary: "Fast-lane to ozone build" It was very hard > to resist...) > > The two slowest part of Ozone build as of now: > # The (multiple) shading of ozonefs > # And the frontend build/obfuscation of ozone recon > [~aengineer] suggested to introduce options to skip them as they are not > required for the build all the time. > This patch introduces '-DskipRecon' and '-DskipShade' options to provide a > faster way to create a *partial* build. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
