[jira] [Updated] (HDDS-3639) Maintain FileHandle Information in OMMetadataManager
[ https://issues.apache.org/jira/browse/HDDS-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3639: - Labels: pull-request-available (was: ) > Maintain FileHandle Information in OMMetadataManager > > > Key: HDDS-3639 > URL: https://issues.apache.org/jira/browse/HDDS-3639 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Filesystem >Reporter: Prashant Pogde >Assignee: Prashant Pogde >Priority: Major > Labels: pull-request-available > > Maintain FileHandle Information in OMMetadataManager. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] prashantpogde opened a new pull request #1004: HDDS-3639. Maintain FileHandle Information in OMMetadataManager.
prashantpogde opened a new pull request #1004: URL: https://github.com/apache/hadoop-ozone/pull/1004 ## What changes were proposed in this pull request? Maintain FileHandle Information in OMMetadataManager ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3639 ## How was this patch tested? Clean build and UTs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3639) Maintain FileHandle Information in OMMetadataManager
[ https://issues.apache.org/jira/browse/HDDS-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Pogde updated HDDS-3639: - Summary: Maintain FileHandle Information in OMMetadataManager (was: Maintain FileHandle Information in OMMetadataManager tables) > Maintain FileHandle Information in OMMetadataManager > > > Key: HDDS-3639 > URL: https://issues.apache.org/jira/browse/HDDS-3639 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Filesystem >Reporter: Prashant Pogde >Assignee: Prashant Pogde >Priority: Major > > Maintain FileHandle Information in OMMetadataManager. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3692) Consider avoiding stream/map/sum in write hotpath
[
https://issues.apache.org/jira/browse/HDDS-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123342#comment-17123342
]
Nanda kumar commented on HDDS-3692:
---
Thanks [~rajesh.balamohan] for the contribution. Added you as a contributor in
HDDS.
> Consider avoiding stream/map/sum in write hotpath
> -
>
> Key: HDDS-3692
> URL: https://issues.apache.org/jira/browse/HDDS-3692
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Filesystem
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Critical
> Labels: Triaged, performance, pull-request-available
> Fix For: 0.6.0
>
> Attachments: HDDS-3692.1.patch, image-2020-06-01-11-08-08-711.png
>
>
> [https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BufferPool.java#L113]
>
> {noformat}
> public long computeBufferData() {
> return bufferList.stream().mapToInt(ChunkBuffer::position)
> .sum();
> }
> {noformat}
> [https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java#L256]
> [https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java#L264]
> Getting back to regular iteration should fix the issue. This causes 3x CPU
> expensive when compared to HDFS flow.
>
> !image-2020-06-01-11-08-08-711.png|width=777,height=564!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3692) Consider avoiding stream/map/sum in write hotpath
[
https://issues.apache.org/jira/browse/HDDS-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nanda kumar reassigned HDDS-3692:
-
Assignee: Rajesh Balamohan
> Consider avoiding stream/map/sum in write hotpath
> -
>
> Key: HDDS-3692
> URL: https://issues.apache.org/jira/browse/HDDS-3692
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Filesystem
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Critical
> Labels: Triaged, performance, pull-request-available
> Fix For: 0.6.0
>
> Attachments: HDDS-3692.1.patch, image-2020-06-01-11-08-08-711.png
>
>
> [https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BufferPool.java#L113]
>
> {noformat}
> public long computeBufferData() {
> return bufferList.stream().mapToInt(ChunkBuffer::position)
> .sum();
> }
> {noformat}
> [https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java#L256]
> [https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java#L264]
> Getting back to regular iteration should fix the issue. This causes 3x CPU
> expensive when compared to HDFS flow.
>
> !image-2020-06-01-11-08-08-711.png|width=777,height=564!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3196) New PipelineManager interface to persist to RatisServer
[ https://issues.apache.org/jira/browse/HDDS-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nanda kumar updated HDDS-3196: -- Target Version/s: (was: 0.6.0) > New PipelineManager interface to persist to RatisServer > --- > > Key: HDDS-3196 > URL: https://issues.apache.org/jira/browse/HDDS-3196 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Li Cheng >Assignee: Li Cheng >Priority: Major > Labels: Triaged, pull-request-available > > This applies to DestroyPipeline as well createPipeline -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3196) New PipelineManager interface to persist to RatisServer
[ https://issues.apache.org/jira/browse/HDDS-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nanda kumar updated HDDS-3196: -- Resolution: Fixed Status: Resolved (was: Patch Available) > New PipelineManager interface to persist to RatisServer > --- > > Key: HDDS-3196 > URL: https://issues.apache.org/jira/browse/HDDS-3196 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Li Cheng >Assignee: Li Cheng >Priority: Major > Labels: Triaged, pull-request-available > > This applies to DestroyPipeline as well createPipeline -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2371) Print out the ozone version during the startup instead of hadoop version
[
https://issues.apache.org/jira/browse/HDDS-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Chitlangia updated HDDS-2371:
Fix Version/s: 0.6.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks [~elek] for reporting, [~adoroszlai] for the contribution.
> Print out the ozone version during the startup instead of hadoop version
>
>
> Key: HDDS-2371
> URL: https://issues.apache.org/jira/browse/HDDS-2371
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Marton Elek
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: Triaged, pull-request-available
> Fix For: 0.6.0
>
>
> Ozone components printing out the current version during the startup:
>
> {code:java}
> STARTUP_MSG: Starting StorageContainerManager
> STARTUP_MSG: host = om/10.8.0.145
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 3.2.0
> STARTUP_MSG: build = https://github.com/apache/hadoop.git -r
> e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on
> 2019-01-{code}
> But as it's visible the build / compiled information is about hadoop not
> about hadoop-ozone.
> (And personally I prefer to use a github compatible url instead of the SVN
> style -r. Something like:
> {code:java}
> STARTUP_MSG: build =
> https://github.com/apache/hadoop-ozone/commit/8541c5694efebb58f53cf4665d3e4e6e4a12845c
> ; compiled by '' on ...{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia merged pull request #869: HDDS-2371. Print Ozone version during startup instead of Hadoop version
dineshchitlangia merged pull request #869: URL: https://github.com/apache/hadoop-ozone/pull/869 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3692) Consider avoiding stream/map/sum in write hotpath
[
https://issues.apache.org/jira/browse/HDDS-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Chitlangia updated HDDS-3692:
Fix Version/s: 0.6.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Consider avoiding stream/map/sum in write hotpath
> -
>
> Key: HDDS-3692
> URL: https://issues.apache.org/jira/browse/HDDS-3692
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Filesystem
>Reporter: Rajesh Balamohan
>Priority: Critical
> Labels: Triaged, performance, pull-request-available
> Fix For: 0.6.0
>
> Attachments: HDDS-3692.1.patch, image-2020-06-01-11-08-08-711.png
>
>
> [https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BufferPool.java#L113]
>
> {noformat}
> public long computeBufferData() {
> return bufferList.stream().mapToInt(ChunkBuffer::position)
> .sum();
> }
> {noformat}
> [https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java#L256]
> [https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java#L264]
> Getting back to regular iteration should fix the issue. This causes 3x CPU
> expensive when compared to HDFS flow.
>
> !image-2020-06-01-11-08-08-711.png|width=777,height=564!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia merged pull request #1003: HDDS-3692: Consider avoiding stream/map/sum in write hotpath
dineshchitlangia merged pull request #1003: URL: https://github.com/apache/hadoop-ozone/pull/1003 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3648) Volume quota is not enforced
[ https://issues.apache.org/jira/browse/HDDS-3648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17121527#comment-17121527 ] Simon Su commented on HDDS-3648: Thanks Dinesh, I will continuously track this issue. > Volume quota is not enforced > > > Key: HDDS-3648 > URL: https://issues.apache.org/jira/browse/HDDS-3648 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.5.0 >Reporter: Simon Su >Assignee: Simon Su >Priority: Major > Labels: Triaged > Attachments: image-2020-05-25-11-04-59-048.png, > image-2020-05-25-11-06-25-030.png, image-2020-05-25-11-06-50-072.png > > > As pic show below, even though I have set quota for volume, a big file which > exceed the quota can still be uploaded. > > 1. Create volume with quota. > !image-2020-05-25-11-04-59-048.png! > > 2. create bucket1 under volume and upload a 12MB file. > ozone sh bucket create /vo1/bucket1 > > !image-2020-05-25-11-06-25-030.png! > > !image-2020-05-25-11-06-50-072.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3648) Volume quota is not enforced
[ https://issues.apache.org/jira/browse/HDDS-3648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3648: Target Version/s: 0.6.0 Labels: Triaged (was: ) > Volume quota is not enforced > > > Key: HDDS-3648 > URL: https://issues.apache.org/jira/browse/HDDS-3648 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.5.0 >Reporter: Simon Su >Assignee: Simon Su >Priority: Major > Labels: Triaged > Attachments: image-2020-05-25-11-04-59-048.png, > image-2020-05-25-11-06-25-030.png, image-2020-05-25-11-06-50-072.png > > > As pic show below, even though I have set quota for volume, a big file which > exceed the quota can still be uploaded. > > 1. Create volume with quota. > !image-2020-05-25-11-04-59-048.png! > > 2. create bucket1 under volume and upload a 12MB file. > ozone sh bucket create /vo1/bucket1 > > !image-2020-05-25-11-06-25-030.png! > > !image-2020-05-25-11-06-50-072.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3648) Volume quota is not enforced
[ https://issues.apache.org/jira/browse/HDDS-3648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17121521#comment-17121521 ] Dinesh Chitlangia commented on HDDS-3648: - [~simonss] Thanks for reporting this problem. We discussed about this in today's community call and its confirmed that the volume quota check is never performed. cc: [~elek] , [~msingh] > Volume quota is not enforced > > > Key: HDDS-3648 > URL: https://issues.apache.org/jira/browse/HDDS-3648 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.5.0 >Reporter: Simon Su >Assignee: Simon Su >Priority: Major > Attachments: image-2020-05-25-11-04-59-048.png, > image-2020-05-25-11-06-25-030.png, image-2020-05-25-11-06-50-072.png > > > As pic show below, even though I have set quota for volume, a big file which > exceed the quota can still be uploaded. > > 1. Create volume with quota. > !image-2020-05-25-11-04-59-048.png! > > 2. create bucket1 under volume and upload a 12MB file. > ozone sh bucket create /vo1/bucket1 > > !image-2020-05-25-11-06-25-030.png! > > !image-2020-05-25-11-06-50-072.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3648) Volume quota is not enforced
[ https://issues.apache.org/jira/browse/HDDS-3648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia updated HDDS-3648: Summary: Volume quota is not enforced (was: Volume quota value not work) > Volume quota is not enforced > > > Key: HDDS-3648 > URL: https://issues.apache.org/jira/browse/HDDS-3648 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.5.0 >Reporter: Simon Su >Assignee: Simon Su >Priority: Major > Attachments: image-2020-05-25-11-04-59-048.png, > image-2020-05-25-11-06-25-030.png, image-2020-05-25-11-06-50-072.png > > > As pic show below, even though I have set quota for volume, a big file which > exceed the quota can still be uploaded. > > 1. Create volume with quota. > !image-2020-05-25-11-04-59-048.png! > > 2. create bucket1 under volume and upload a 12MB file. > ozone sh bucket create /vo1/bucket1 > > !image-2020-05-25-11-06-25-030.png! > > !image-2020-05-25-11-06-50-072.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rbalamohan opened a new pull request #1003: HDDS-3692: Consider avoiding stream/map/sum in write hotpath
rbalamohan opened a new pull request #1003: URL: https://github.com/apache/hadoop-ozone/pull/1003 ## What changes were proposed in this pull request? HDDS-3692: Consider avoiding stream/map/sum in write hotpath. Stream in write hotpath makes it highly CPU intensive. Fixing it in this PR. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3692 ## How was this patch tested? (Please explain how this patch was tested. Ex: unit tests, manual tests) (If this patch involves UI changes, please attach a screen-shot; otherwise, remove this) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3692) Consider avoiding stream/map/sum in write hotpath
[
https://issues.apache.org/jira/browse/HDDS-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-3692:
-
Labels: Triaged performance pull-request-available (was: Triaged
performance)
> Consider avoiding stream/map/sum in write hotpath
> -
>
> Key: HDDS-3692
> URL: https://issues.apache.org/jira/browse/HDDS-3692
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Filesystem
>Reporter: Rajesh Balamohan
>Priority: Critical
> Labels: Triaged, performance, pull-request-available
> Attachments: HDDS-3692.1.patch, image-2020-06-01-11-08-08-711.png
>
>
> [https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BufferPool.java#L113]
>
> {noformat}
> public long computeBufferData() {
> return bufferList.stream().mapToInt(ChunkBuffer::position)
> .sum();
> }
> {noformat}
> [https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java#L256]
> [https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java#L264]
> Getting back to regular iteration should fix the issue. This causes 3x CPU
> expensive when compared to HDFS flow.
>
> !image-2020-06-01-11-08-08-711.png|width=777,height=564!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2969) Implement ofs://: Add contract test
[ https://issues.apache.org/jira/browse/HDDS-2969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-2969: Target Version/s: 0.6.0 Labels: Triaged pull-request-available (was: pull-request-available) > Implement ofs://: Add contract test > --- > > Key: HDDS-2969 > URL: https://issues.apache.org/jira/browse/HDDS-2969 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: Triaged, pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-54) Add support for KSM --expunge command
[ https://issues.apache.org/jira/browse/HDDS-54?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-54: -- Target Version/s: 0.7.0 > Add support for KSM --expunge command > - > > Key: HDDS-54 > URL: https://issues.apache.org/jira/browse/HDDS-54 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Affects Versions: 0.2.1 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-12742-HDFS-7240.001.patch, > HDFS-12742-HDFS-7240.002.patch, HDFS-12742-HDFS-7240.003.patch > > > KSM --expunge will delete all the data from the data nodes for all the keys > in the KSM db. > User will have no control over the deletion. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2371) Print out the ozone version during the startup instead of hadoop version
[
https://issues.apache.org/jira/browse/HDDS-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-2371:
Target Version/s: 0.6.0
Labels: Triaged pull-request-available (was:
pull-request-available)
> Print out the ozone version during the startup instead of hadoop version
>
>
> Key: HDDS-2371
> URL: https://issues.apache.org/jira/browse/HDDS-2371
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Marton Elek
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: Triaged, pull-request-available
>
> Ozone components printing out the current version during the startup:
>
> {code:java}
> STARTUP_MSG: Starting StorageContainerManager
> STARTUP_MSG: host = om/10.8.0.145
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 3.2.0
> STARTUP_MSG: build = https://github.com/apache/hadoop.git -r
> e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on
> 2019-01-{code}
> But as it's visible the build / compiled information is about hadoop not
> about hadoop-ozone.
> (And personally I prefer to use a github compatible url instead of the SVN
> style -r. Something like:
> {code:java}
> STARTUP_MSG: build =
> https://github.com/apache/hadoop-ozone/commit/8541c5694efebb58f53cf4665d3e4e6e4a12845c
> ; compiled by '' on ...{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3090) Fix logging in OMFileRequest and OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3090: Target Version/s: 0.6.0 Labels: Triaged pull-request-available (was: pull-request-available) > Fix logging in OMFileRequest and OzoneManager > - > > Key: HDDS-3090 > URL: https://issues.apache.org/jira/browse/HDDS-3090 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Trivial > Labels: Triaged, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HDDS-2940 introduced a INFO level log in > hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java > This needs to be a TRACE because it occurs in the regular file create path. > Also, trace logs introduced in OzoneManager and OMFileRequest.java need to be > parameterized. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3111) Add unit test for container replication behavior under different container placement policy
[ https://issues.apache.org/jira/browse/HDDS-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3111: Target Version/s: 0.7.0 > Add unit test for container replication behavior under different container > placement policy > --- > > Key: HDDS-3111 > URL: https://issues.apache.org/jira/browse/HDDS-3111 > Project: Hadoop Distributed Data Store > Issue Type: Test >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently, the unit test for ReplicationManager only tested for container > state change and container placement policy only focus on the policy > algorithm. > And we lack of one integration unit test for testing container replication > behavior under different container placement policy. Including some corner > cases, like not enough candidate node, fallback cases in rack awareness > policy. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3196) New PipelineManager interface to persist to RatisServer
[ https://issues.apache.org/jira/browse/HDDS-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3196: Target Version/s: 0.6.0 Labels: Triaged pull-request-available (was: pull-request-available) > New PipelineManager interface to persist to RatisServer > --- > > Key: HDDS-3196 > URL: https://issues.apache.org/jira/browse/HDDS-3196 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Li Cheng >Assignee: Li Cheng >Priority: Major > Labels: Triaged, pull-request-available > > This applies to DestroyPipeline as well createPipeline -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3367) Trash Support for Ozone FileSystem
[ https://issues.apache.org/jira/browse/HDDS-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3367: Target Version/s: 0.6.0 > Trash Support for Ozone FileSystem > -- > > Key: HDDS-3367 > URL: https://issues.apache.org/jira/browse/HDDS-3367 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Filesystem >Affects Versions: 0.6.0 >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Minor > Labels: TriagePending > Attachments: OzoneFsTrash.patch > > > The aim of this jira to add the trash feature capability in ozone that will > help prevent accidental deletion of files and directories. When a file gets > deleted in Ozone, the file is not immediately expelled from Ozone but will be > moved to trash. Similarly as in HDFS, to immediately delete a file, a > skipTrash option will be provided. The jira also aims to integrate the Ozone > FS Trash to the Trash API developed as a part of HDDS-2416. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3642) Stop/Pause Background services while replacing OM DB with checkpoint from Leader
[
https://issues.apache.org/jira/browse/HDDS-3642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-3642:
-
Labels: pull-request-available (was: )
> Stop/Pause Background services while replacing OM DB with checkpoint from
> Leader
>
>
> Key: HDDS-3642
> URL: https://issues.apache.org/jira/browse/HDDS-3642
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Critical
> Labels: pull-request-available
>
> When a follower OM needs to replace its DB with a checkpoint from Leader (to
> catch up on the transactions), it should pause or stop services which read/
> write to the DB.
> During OM HA testing, found that OM could crash with JVM error on RocksDB.
> This happened because KeyDeletingService was trying to access a memory which
> is already freed up.
> {code:java}
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x7f19de835af0, pid=1389, tid=1712
> #
> # JRE version: OpenJDK Runtime Environment (11.0.6+10) (build 11.0.6+10-LTS)
> # Java VM: OpenJDK 64-Bit Server VM (11.0.6+10-LTS, mixed mode, sharing,
> tiered, compressed oops, concurrent mark sweep gc, linux-amd64)
> # Problematic frame:
> # C [librocksdbjni10001996641283911793.so+0x1aeaf0]
> Java_org_rocksdb_RocksIterator_seekToFirst0+0x0
> #
> # Core dump will be written. Default location: Core dumps may be processed
> with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to
> /opt/core.1389)
> #
> # An error report file with more information is saved as:
> # /opt/hs_err_pid1389.log
> {code}
> From the hs_error log file:
> {code:java}
> --- T H R E A D ---Current thread
> (0x011a4000): JavaThread "KeyDeletingService#1" daemon
> [_thread_in_native, id=1712,
> stack(0x7f19d2443000,0x7f19d2544000)]Stack:
> [0x7f19d2443000,0x7f19d2544000], sp=0x7f19d2541e78, free
> space=1019k
> Native frames: (J=compiled Java code, A=aot compiled Java code,
> j=interpreted, Vv=VM code, C=native code)
> C [librocksdbjni10001996641283911793.so+0x1aeaf0]
> Java_org_rocksdb_RocksIterator_seekToFirst0+0x0
> j org.rocksdb.AbstractRocksIterator.seekToFirst()V+26
> j
> org.apache.hadoop.hdds.utils.db.RDBStoreIterator.(Lorg/rocksdb/RocksIterator;)V+13
> j
> org.apache.hadoop.hdds.utils.db.RDBTable.iterator()Lorg/apache/hadoop/hdds/utils/db/TableIterator;+30
> j
> org.apache.hadoop.hdds.utils.db.TypedTable.iterator()Lorg/apache/hadoop/hdds/utils/db/TableIterator;+4
> j
> org.apache.hadoop.ozone.om.OmMetadataManagerImpl.getPendingDeletionKeys(I)Ljava/util/List;+8
> j
> org.apache.hadoop.ozone.om.KeyManagerImpl.getPendingDeletionKeys(I)Ljava/util/List;+5
> j
> org.apache.hadoop.ozone.om.KeyDeletingService$KeyDeletingTask.call()Lorg/apache/hadoop/hdds/utils/BackgroundTaskResult;+39
> j
> org.apache.hadoop.ozone.om.KeyDeletingService$KeyDeletingTask.call()Ljava/lang/Object;+1
> J 4791 c1 java.util.concurrent.FutureTask.run()V java.base@11.0.6 (123 bytes)
> @ 0x7f19f0c7b414 [0x7f19f0c7ad20+0x06f4]
> J 4802 c1
> java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object;
> java.base@11.0.6 (14 bytes) @ 0x7f19f0c87214
> [0x7f19f0c870e0+0x0134]
> J 4791 c1 java.util.concurrent.FutureTask.run()V java.base@11.0.6 (123 bytes)
> @ 0x7f19f0c7b414 [0x7f19f0c7ad20+0x06f4]
> J 4802 c1
> java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object;
> java.base@11.0.6 (14 bytes) @ 0x7f19f0c87214
> [0x7f19f0c870e0+0x0134]
> J 4791 c1 java.util.concurrent.FutureTask.run()V java.base@11.0.6 (123 bytes)
> @ 0x7f19f0c7b414 [0x7f19f0c7ad20+0x06f4]
> J 4954 c1
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V
> java.base@11.0.6 (57 bytes) @ 0x7f19f0cfe10c
> [0x7f19f0cfde40+0x02cc]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] hanishakoneru opened a new pull request #1002: HDDS-3642. Stop/Pause Background services while replacing OM DB with checkpoint from Leader
hanishakoneru opened a new pull request #1002: URL: https://github.com/apache/hadoop-ozone/pull/1002 ## What changes were proposed in this pull request? When a follower OM needs to replace its DB with a checkpoint from Leader (to catch up on the transactions), it should pause or stop services which read/ write to the DB. During OM HA testing, found that OM could crash with JVM error on RocksDB. This happened because KeyDeletingService was trying to access a memory which is already freed up. Please see Jira link below for error logs. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3642 ## How was this patch tested? Tested on a local docker cluster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3476) Use persisted transaction info during OM startup in OM StateMachine
[ https://issues.apache.org/jira/browse/HDDS-3476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3476: Target Version/s: 0.6.0 Labels: Triaged pull-request-available (was: pull-request-available) > Use persisted transaction info during OM startup in OM StateMachine > --- > > Key: HDDS-3476 > URL: https://issues.apache.org/jira/browse/HDDS-3476 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: Triaged, pull-request-available > > HDDS-3475 persisted transaction info into DB. This Jira is to use > transactionInfo persisted to DB during OM startup. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3574) Implement ofs://: Override getTrashRoot
[
https://issues.apache.org/jira/browse/HDDS-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3574:
Target Version/s: 0.6.0
Labels: Triaged pull-request-available (was:
pull-request-available)
> Implement ofs://: Override getTrashRoot
> ---
>
> Key: HDDS-3574
> URL: https://issues.apache.org/jira/browse/HDDS-3574
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: Triaged, pull-request-available
>
> [~pifta] found if we delete file with Hadoop shell, namely {{hadoop fs -rm}},
> without {{-skipTrash}} option, the operation would fail in OFS due to the
> client is renaming the file to {{/user//.Trash/}} because renaming
> across different buckets is not allowed in Ozone. (Unless the file happens to
> be under {{/user//}}, apparently.)
> We could override {{getTrashRoot()}} in {{BasicOzoneFileSystem}} to a dir
> under the same bucket to mitigate the problem. Thanks [~umamaheswararao] for
> the suggestion.
> This raises one more problem though: We need to implement trash clean up on
> OM. Opened HDDS-3575 for this.
> CC [~arp] [~bharat]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3596) Clean up unused code after HDDS-2940 and HDDS-2942
[
https://issues.apache.org/jira/browse/HDDS-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3596:
Target Version/s: 0.6.0
Labels: Triaged pull-request-available (was:
pull-request-available)
> Clean up unused code after HDDS-2940 and HDDS-2942
> --
>
> Key: HDDS-3596
> URL: https://issues.apache.org/jira/browse/HDDS-3596
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: Triaged, pull-request-available
>
> It seems some snippets of code should be removed as HDDS-2940 is committed.
> Update: Pending HDDS-2942 commit before this can be committed.
> For example
> [this|https://github.com/apache/hadoop-ozone/blob/ffb340e32460ccaa2eae557f0bb71fb90d7ebc7a/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java#L495-L499]:
> {code:java|title=BasicOzoneFileSystem#delete}
> if (result) {
> // If this delete operation removes all files/directories from the
> // parent directory, then an empty parent directory must be created.
> createFakeParentDirectory(f);
> }
> {code}
> (Found at
> https://github.com/apache/hadoop-ozone/pull/906#discussion_r424873030)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3599) Implement ofs://: Add contract test for HA
[ https://issues.apache.org/jira/browse/HDDS-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3599: Target Version/s: 0.6.0 Labels: Triaged pull-request-available (was: pull-request-available) > Implement ofs://: Add contract test for HA > -- > > Key: HDDS-3599 > URL: https://issues.apache.org/jira/browse/HDDS-3599 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: Triaged, pull-request-available > > Add contract tests for HA as well. > Since adding HA contract tests will be another ~10 new classes. [~xyao] and I > decided to put HA OFS contract tests in another jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3614) Remove S3Table from OmMetadataManager
[ https://issues.apache.org/jira/browse/HDDS-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3614: Target Version/s: 0.6.0 Labels: Triaged pull-request-available (was: pull-request-available) > Remove S3Table from OmMetadataManager > - > > Key: HDDS-3614 > URL: https://issues.apache.org/jira/browse/HDDS-3614 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: Triaged, pull-request-available > > Remove the S3 table, after HDDS-3385, we don't have any use case for S3Table. > We can remove this table from OmMetadataManager. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3644) Failed to delete chunk file due to chunk size mismatch
[
https://issues.apache.org/jira/browse/HDDS-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3644:
Target Version/s: 0.6.0
Labels: Triaged pull-request-available (was:
pull-request-available)
> Failed to delete chunk file due to chunk size mismatch
> --
>
> Key: HDDS-3644
> URL: https://issues.apache.org/jira/browse/HDDS-3644
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Sammi Chen
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: Triaged, pull-request-available
>
> LOGs
> {noformat}
> 2020-05-19 13:45:30,493 [BlockDeletingService#8] WARN
> org.apache.hadoop.ozone.container.keyvalue.impl.FilePerChunkStrategy: Chunk
> file doe not exist. chunk info
> :ChunkInfo{chunkName='104079328540607246_chunk_1, offset=0, len=4194304}
> 2020-05-19 13:45:30,493 [BlockDeletingService#8] ERROR
> org.apache.hadoop.ozone.container.keyvalue.impl.FilePerChunkStrategy: Not
> Supported Operation. Trying to delete a chunk that is in shared file. chunk
> info : ChunkInfo{chunkName='104079328540607246_chunk_2, offset=4194304,
> len=1048576}
> 2020-05-19 13:45:30,494 [BlockDeletingService#8] ERROR
> org.apache.hadoop.ozone.container.keyvalue.statemachine.background.BlockDeletingService:
> Failed to delete files for block #deleting#104079328540607246
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
>
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> Not Supported Operation. Trying to delete a chunk that is in shared file.
> chunk info : ChunkInfo{chunkName='104079328540607246_chunk_2, offset=4194304,
> len=1048576}
> at
> org.apache.hadoop.ozone.container.keyvalue.impl.FilePerChunkStrategy.deleteChunks(FilePerChunkStrategy.java:286)
> at
> org.apache.hadoop.ozone.container.keyvalue.impl.ChunkManagerDispatcher.deleteChunks(ChunkManagerDispatcher.java:111)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.deleteBlock(KeyValueHandler.java:1043)
> at
> org.apache.hadoop.ozone.container.keyvalue.statemachine.background.BlockDeletingService$BlockDeletingTask.lambda$call$0(BlockDeletingService.java:286)
> Caused by:
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> Not Supported Operation. Trying to delete a chunk that is in shared file.
> chunk info : ChunkInfo{chunkName='104079328540607246_chunk_2, offset=4194304,
> len=1048576}
> {noformat}
> chunk_1 is 4MB and chunk_2 is 1MB in block info.
> chunk_1 doesn't exit(might been deleted successfully) and chunk_2 is 5MB on
> disk.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3615) Call cleanup on tables only when double buffer has transactions related to tables.
[ https://issues.apache.org/jira/browse/HDDS-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3615: Target Version/s: 0.6.0 Labels: Triaged pull-request-available (was: pull-request-available) > Call cleanup on tables only when double buffer has transactions related to > tables. > -- > > Key: HDDS-3615 > URL: https://issues.apache.org/jira/browse/HDDS-3615 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: Triaged, pull-request-available > > For volume/bucket table currently it is full cache, and we need to cleanup > entries only when they are marked for delete. > So, it is unnecessary to call cleanup and waste the CPU resources on OM. > Similarly for other tables, when the double buffer has transaction entries > that touch those tables, then only call cleanup. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #814: HDDS-3286. BasicOzoneFileSystem support batchDelete.
xiaoyuyao commented on a change in pull request #814:
URL: https://github.com/apache/hadoop-ozone/pull/814#discussion_r433519025
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMBatchKeyDeleteRequest.java
##
@@ -0,0 +1,226 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.request.key;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import org.apache.hadoop.hdds.utils.db.cache.CacheKey;
+import org.apache.hadoop.hdds.utils.db.cache.CacheValue;
+import org.apache.hadoop.ozone.audit.AuditLogger;
+import org.apache.hadoop.ozone.audit.OMAction;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OMMetrics;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.exceptions.OMReplayException;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.ratis.utils.OzoneManagerDoubleBufferHelper;
+import org.apache.hadoop.ozone.om.request.util.OmResponseUtil;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.ozone.om.response.key.OMKeyDeleteResponse;
+import org.apache.hadoop.ozone.om.response.key.OMBatchKeyDeleteResponse;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.*;
+import org.apache.hadoop.ozone.security.acl.IAccessAuthorizer;
+import org.apache.hadoop.ozone.security.acl.OzoneObj;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.*;
+
+import static
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.KEY_NOT_FOUND;
+import static
org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+
+/**
+ * Handles DeleteKey request.
+ */
+public class OMBatchKeyDeleteRequest extends OMKeyRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OMBatchKeyDeleteRequest.class);
+
+ public OMBatchKeyDeleteRequest(OMRequest omRequest) {
+super(omRequest);
+ }
+
+ @Override
+ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
+DeleteBatchKeyRequest deleteKeyRequest =
+getOmRequest().getDeleteBatchKeyRequest();
+Preconditions.checkNotNull(deleteKeyRequest);
+List newKeyArgsList = new ArrayList<>();
+for (KeyArgs keyArgs : deleteKeyRequest.getKeyArgsList()) {
+ newKeyArgsList.add(
+ keyArgs.toBuilder().setModificationTime(Time.now()).build());
+}
+DeleteBatchKeyRequest newDeleteKeyRequest = DeleteBatchKeyRequest
+.newBuilder().addAllKeyArgs(newKeyArgsList).build();
+
+return getOmRequest().toBuilder()
+.setDeleteBatchKeyRequest(newDeleteKeyRequest)
+.setUserInfo(getUserInfo()).build();
+ }
+
+ @Override
+ @SuppressWarnings("methodlength")
+ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
+ long trxnLogIndex, OzoneManagerDoubleBufferHelper omDoubleBufferHelper) {
+DeleteBatchKeyRequest deleteKeyRequest =
+getOmRequest().getDeleteBatchKeyRequest();
+
+List deleteKeyArgsList = deleteKeyRequest.getKeyArgsList();
+IOException exception = null;
+OMClientResponse omClientResponse = null;
+Result result = null;
+
+OMMetrics omMetrics = ozoneManager.getMetrics();
+omMetrics.incNumKeyDeletes();
+List omKeyInfoList = new ArrayList<>();
+Set acquiredLockSet = new HashSet<>();
+Map auditMap = null;
+String volumeName = "";
+String bucketName = "";
+String keyName = "";
+
+AuditLogger auditLogger = ozoneManager.getAuditLogger();
+OzoneManagerProtocolProtos.UserInfo userInfo =
+getOmRequest().getUserInfo();
+
+OMResponse.Builder omResponse = OmResponseUtil.getOMResponseBuilder(
+getOmRequest());
+OMMetadataManager omMetadataManager = ozoneManager.getMetadataManager();
+try {
+ for (KeyArgs deleteKeyArgs : deleteKeyArgsList) {
+
[GitHub] [hadoop-ozone] hanishakoneru commented on pull request #925: HDDS-3586. OM HA can be started with 3 isolated LEADER instead of one OM ring
hanishakoneru commented on pull request #925: URL: https://github.com/apache/hadoop-ozone/pull/925#issuecomment-637153981 Hi @elek, did you get a chance to check this again? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3692) Consider avoiding stream/map/sum in write hotpath
[
https://issues.apache.org/jira/browse/HDDS-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3692:
Labels: Triaged performance (was: performance)
> Consider avoiding stream/map/sum in write hotpath
> -
>
> Key: HDDS-3692
> URL: https://issues.apache.org/jira/browse/HDDS-3692
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Filesystem
>Reporter: Rajesh Balamohan
>Priority: Critical
> Labels: Triaged, performance
> Attachments: HDDS-3692.1.patch, image-2020-06-01-11-08-08-711.png
>
>
> [https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BufferPool.java#L113]
>
> {noformat}
> public long computeBufferData() {
> return bufferList.stream().mapToInt(ChunkBuffer::position)
> .sum();
> }
> {noformat}
> [https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java#L256]
> [https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java#L264]
> Getting back to regular iteration should fix the issue. This causes 3x CPU
> expensive when compared to HDFS flow.
>
> !image-2020-06-01-11-08-08-711.png|width=777,height=564!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3674) add a ozone-style.xml that can be imported into IDEA.
[ https://issues.apache.org/jira/browse/HDDS-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDDS-3674. - Fix Version/s: 0.6.0 Resolution: Fixed I've committed this. Thanks for the contribution [~micahzhao]. > add a ozone-style.xml that can be imported into IDEA. > - > > Key: HDDS-3674 > URL: https://issues.apache.org/jira/browse/HDDS-3674 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Tools >Reporter: mingchao zhao >Assignee: mingchao zhao >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > > Add a ozone-style.xml, Based on the current requirements of ozone > checkstyle. > After importing into IDEA, you can format the code as required by ozone. > Avoiding the checkstyle problem in CI, this can improve our development > efficiency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] arp7 merged pull request #978: HDDS-3674. add a ozone-style.xml that can be imported into IDEA.
arp7 merged pull request #978: URL: https://github.com/apache/hadoop-ozone/pull/978 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3669) SCM Infinite loop in BlockManagerImpl.allocateBlock
[
https://issues.apache.org/jira/browse/HDDS-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3669:
Target Version/s: 0.6.0
Labels: TriagePending (was: )
> SCM Infinite loop in BlockManagerImpl.allocateBlock
> ---
>
> Key: HDDS-3669
> URL: https://issues.apache.org/jira/browse/HDDS-3669
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.6.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: TriagePending
>
> The following step can reproduce this issue
> - A new ozone cluster with only a factor three pipeline
> - put a big file(1G) into cluster, during the put process, we kill the
> leader datanode of this pipeline.
> The put command will hang, the following log will fill the scm log file.
> 2020-05-27 17:32:46,988 [IPC Server handler 23 on default port 9863] WARN
> org.apache.hadoop.hdds.scm.container.SCMContainerManager: Container
> allocation failed for pipeline=Pipeline[ Id:
> bf7dd356-2d97-4b2a-8a81-e2ddd25bc5a1, Nodes:
> e859cad9-c7f6-451a-a039-af06103aa978{ip: 127.0.0.1, host: localhost,
> networkLocation: /default-rack, certSerialId:
> null}1cd2bf20-a791-42a0-b4cd-b26d995cb8eb{ip: 127.0.0.1, host: localhost,
> networkLocation: /default-rack, certSerialId:
> null}0827f3bb-0d94-435a-a157-4db2c84cdedf{ip: 127.0.0.1, host: localhost,
> networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:3,
> State:OPEN, leaderId:0827f3bb-0d94-435a-a157-4db2c84cdedf,
> CreationTimestamp2020-05-27T08:05:36.590Z] requiredSize=268435456 {}
> org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException:
> PipelineID=bf7dd356-2d97-4b2a-8a81-e2ddd25bc5a1 not found
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.getContainers(PipelineStateMap.java:301)
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager.getContainers(PipelineStateManager.java:95)
> at
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.getContainersInPipeline(SCMPipelineManager.java:360)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getContainersForOwner(SCMContainerManager.java:507)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.getMatchingContainer(SCMContainerManager.java:428)
> at
> org.apache.hadoop.hdds.scm.block.BlockManagerImpl.allocateBlock(BlockManagerImpl.java:230)
> at
> org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer.allocateBlock(SCMBlockProtocolServer.java:190)
> at
> org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.allocateScmBlock(ScmBlockLocationProtocolServerSideTranslatorPB.java:167)
> at
> org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.processMessage(ScmBlockLocationProtocolServerSideTranslatorPB.java:119)
> at
> org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:74)
> at
> org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:100)
> at
> org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:13303)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:999)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:927)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2915)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3668) OzoneManager start fails with RocksDB error on downgrade to older version.
[
https://issues.apache.org/jira/browse/HDDS-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3668:
Fix Version/s: (was: 0.6.0)
Target Version/s: 0.6.0
Labels: Triaged pull-request-available upgrade (was:
pull-request-available upgrade)
> OzoneManager start fails with RocksDB error on downgrade to older version.
> --
>
> Key: HDDS-3668
> URL: https://issues.apache.org/jira/browse/HDDS-3668
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Affects Versions: 0.6.0
>Reporter: Bharat Viswanadham
>Assignee: Aravindan Vijayan
>Priority: Critical
> Labels: Triaged, pull-request-available, upgrade
>
> OM start fails with RocksDB error when downgrading to older version that does
> not have all the column families that may have been created in the newer
> version.
> {code}
> java.io.IOException: Failed init RocksDB, db path :
> /tmp/ozone/data/metadata/om.db, exception :org.rocksdb.RocksDBE
> xception You have to open all column families. Column families not opened:
> transactionInfoTable; status : InvalidAr
> gument; message : You have to open all column families. Column families not
> opened: transactionInfoTable
> at
> org.apache.hadoop.hdds.utils.db.RDBStore.toIOException(RDBStore.java:159)
> at org.apache.hadoop.hdds.utils.db.RDBStore.(RDBStore.java:141)
> at
> org.apache.hadoop.hdds.utils.db.DBStoreBuilder.build(DBStoreBuilder.java:181)
> at
> org.apache.hadoop.ozone.om.OmMetadataManagerImpl.start(OmMetadataManagerImpl.java:267)
> at
> org.apache.hadoop.ozone.om.OmMetadataManagerImpl.(OmMetadataManagerImpl.java:164)
> at
> org.apache.hadoop.ozone.om.OzoneManager.instantiateServices(OzoneManager.java:478)
> at
> org.apache.hadoop.ozone.om.OzoneManager.(OzoneManager.java:416)
> at
> org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:884)
> at
> org.apache.hadoop.ozone.om.OzoneManagerStarter$OMStarterHelper.start(OzoneManagerStarter.java:123)
> at
> org.apache.hadoop.ozone.om.OzoneManagerStarter.startOm(OzoneManagerStarter.java:78)
> at
> org.apache.hadoop.ozone.om.OzoneManagerStarter.call(OzoneManagerStarter.java:66)
> at
> org.apache.hadoop.ozone.om.OzoneManagerStarter.call(OzoneManagerStarter.java:37)
> at picocli.CommandLine.execute(CommandLine.java:1173)
> at picocli.CommandLine.access$800(CommandLine.java:141)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:1367)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:1335)
> at
> picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243)
> at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526)
> at picocli.CommandLine.parseWithHandler(CommandLine.java:1465)
> at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:75)
> at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:66)
> at
> org.apache.hadoop.ozone.om.OzoneManagerStarter.main(OzoneManagerStarter.java:50)
> Caused by: org.rocksdb.RocksDBException: You have to open all column
> families. Column families not opened: transact
> ionInfoTable
> at org.rocksdb.RocksDB.open(Native Method)
> at org.rocksdb.RocksDB.open(RocksDB.java:290)
> at org.apache.hadoop.hdds.utils.db.RDBStore.(RDBStore.java:97)
> ... 20 more
> {code}
> Thanks to [~bharat] for reporting this issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3667) If we gracefully stop datanode it would be better to notify scm and recon to unregister
[
https://issues.apache.org/jira/browse/HDDS-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3667:
Target Version/s: 0.7.0
> If we gracefully stop datanode it would be better to notify scm and recon to
> unregister
> ---
>
> Key: HDDS-3667
> URL: https://issues.apache.org/jira/browse/HDDS-3667
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
> Components: Ozone Datanode, Ozone Recon, SCM
>Affects Versions: 0.6.0
>Reporter: maobaolong
>Assignee: Lisheng Sun
>Priority: Minor
>
> if you execute `bin/ozone --daemon stop datanode`, datanode would be better
> if it handle the signal and send the unregister request to scm, then scm
> remove this datanode from scm and reply to the datanode to die.
> It would be better if you provide a admin cli tool to support remove datanode
> manually.
> {code:shell}
> ozone admin datanode remove
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3664) Info command should display the key locations
[
https://issues.apache.org/jira/browse/HDDS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3664:
Target Version/s: 0.7.0
> Info command should display the key locations
> -
>
> Key: HDDS-3664
> URL: https://issues.apache.org/jira/browse/HDDS-3664
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: om, Ozone CLI, Ozone Manager
>Reporter: maobaolong
>Priority: Major
>
> bin/ozone sh key info /myvol/mybucket/NOTICE3.txt
> {
> "volumeName" : "myvol",
> "bucketName" : "mybucket",
> "name" : "NOTICE3.txt",
> "dataSize" : 17540,
> "creationTime" : "2020-05-26T11:49:13.274Z",
> "modificationTime" : "2020-05-26T11:49:15.560Z",
> "replicationType" : "RATIS",
> "replicationFactor" : 3,
> "ozoneKeyLocations" : [ {
> "containerID" : 1,
> "localID" : 104234598612860928,
> "length" : 17540,
> "offset" : 0
> } ],
> "metadata" : { },
> "fileEncryptionInfo" : null
> }
> Through the above response, we just know the containerID, but we want to know
> the container 3 in which pipeline, and the pipeline info.
> Although we can use "ozone admin container list " and "ozone
> admin pipeline list " instead when the pipeline is open, but for
> the closed pipeline, we don't know the replication information.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3659) put a new file to exist key with different factor or type don't update the omkeyinfo
[ https://issues.apache.org/jira/browse/HDDS-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3659: Target Version/s: 0.6.0 Labels: Triaged (was: ) Priority: Critical (was: Major) > put a new file to exist key with different factor or type don't update the > omkeyinfo > > > Key: HDDS-3659 > URL: https://issues.apache.org/jira/browse/HDDS-3659 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: om >Affects Versions: 0.6.0 >Reporter: maobaolong >Priority: Critical > Labels: Triaged > > bin/ozone sh key put -r THREE /myvol/mybucket/NOTICE.txt NOTICE.txt > bin/ozone sh key put -r ONE /myvol/mybucket/NOTICE.txt NOTICE.txt > bin/ozone sh key info /myvol/mybucket/NOTICE.txt NOTICE.txt > it should be ONE -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3655) SCM full of ERROR logs: Failed to create pipeline of type XXX and factor XXX
[
https://issues.apache.org/jira/browse/HDDS-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17121371#comment-17121371
]
Arpit Agarwal commented on HDDS-3655:
-
Hi [~maobaolong] do you have any more info on what was happening before you ran
into this?
> SCM full of ERROR logs: Failed to create pipeline of type XXX and factor XXX
>
>
> Key: HDDS-3655
> URL: https://issues.apache.org/jira/browse/HDDS-3655
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.6.0, 0.7.0
>Reporter: maobaolong
>Priority: Major
> Labels: TriagePending
>
> These log are too noisy.
> {code:java}
> 2020-05-20 08:56:54,839 [RatisPipelineUtilsThread] ERROR
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
> pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed
> because nodes are engaged in other pipelines and every node can only be
> engaged in max 15 pipelines. Required 3. Found 0
> 2020-05-20 08:58:54,839 [RatisPipelineUtilsThread] ERROR
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
> pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of
> factor 1 using 0 nodes. Used 3 nodes. Healthy nodes 3
> 2020-05-20 08:58:54,839 [RatisPipelineUtilsThread] ERROR
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
> pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed
> because nodes are engaged in other pipelines and every node can only be
> engaged in max 15 pipelines. Required 3. Found 0
> 2020-05-20 09:00:54,841 [RatisPipelineUtilsThread] ERROR
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
> pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of
> factor 1 using 0 nodes. Used 3 nodes. Healthy nodes 3
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3655) SCM full of ERROR logs: Failed to create pipeline of type XXX and factor XXX
[
https://issues.apache.org/jira/browse/HDDS-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3655:
Target Version/s: 0.6.0
Labels: TriagePending (was: )
> SCM full of ERROR logs: Failed to create pipeline of type XXX and factor XXX
>
>
> Key: HDDS-3655
> URL: https://issues.apache.org/jira/browse/HDDS-3655
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.6.0, 0.7.0
>Reporter: maobaolong
>Priority: Major
> Labels: TriagePending
>
> These log are too noisy.
> {code:java}
> 2020-05-20 08:56:54,839 [RatisPipelineUtilsThread] ERROR
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
> pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed
> because nodes are engaged in other pipelines and every node can only be
> engaged in max 15 pipelines. Required 3. Found 0
> 2020-05-20 08:58:54,839 [RatisPipelineUtilsThread] ERROR
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
> pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of
> factor 1 using 0 nodes. Used 3 nodes. Healthy nodes 3
> 2020-05-20 08:58:54,839 [RatisPipelineUtilsThread] ERROR
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
> pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed
> because nodes are engaged in other pipelines and every node can only be
> engaged in max 15 pipelines. Required 3. Found 0
> 2020-05-20 09:00:54,841 [RatisPipelineUtilsThread] ERROR
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
> pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of
> factor 1 using 0 nodes. Used 3 nodes. Healthy nodes 3
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3653) Add documentation for Copy key command
[ https://issues.apache.org/jira/browse/HDDS-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3653: Labels: Triaged pull-request-available (was: Triage pull-request-available) > Add documentation for Copy key command > -- > > Key: HDDS-3653 > URL: https://issues.apache.org/jira/browse/HDDS-3653 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone CLI >Affects Versions: 0.6.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Labels: Triaged, pull-request-available > > HDDS-3646 introduced a copy command to copy a key to another one with a > bucket. > The following documents need to be updated to describe the new command usage. > ~/hadoop-ozone/hadoop-hdds/docs/content/shell/KeyCommands.md > ~/hadoop-ozone/hadoop-hdds/docs/content/shell/KeyCommands.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3653) Add documentation for Copy key command
[ https://issues.apache.org/jira/browse/HDDS-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3653: Target Version/s: 0.6.0 Labels: Triage pull-request-available (was: pull-request-available) > Add documentation for Copy key command > -- > > Key: HDDS-3653 > URL: https://issues.apache.org/jira/browse/HDDS-3653 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone CLI >Affects Versions: 0.6.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Labels: Triage, pull-request-available > > HDDS-3646 introduced a copy command to copy a key to another one with a > bucket. > The following documents need to be updated to describe the new command usage. > ~/hadoop-ozone/hadoop-hdds/docs/content/shell/KeyCommands.md > ~/hadoop-ozone/hadoop-hdds/docs/content/shell/KeyCommands.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3620) BasicOzoneFileSystem support batchRename
[ https://issues.apache.org/jira/browse/HDDS-3620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3620: Target Version/s: 0.7.0 > BasicOzoneFileSystem support batchRename > > > Key: HDDS-3620 > URL: https://issues.apache.org/jira/browse/HDDS-3620 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Filesystem >Affects Versions: 0.6.0 >Reporter: mingchao zhao >Priority: Major > > Currently rename folder is to get all the keys, and then rename them one by > one. This makes for poor performance. > HDDS-2939 can able to optimize this part, but at present the hdds-2939 is > slow and still a long way to go. So we optimized the batch operation based on > the current interface. We were able to get better performance with this PR > before the hdds-2939 came in. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3632) HddsDatanodeService cannot be started if HDFS datanode running in same machine with same user.
[
https://issues.apache.org/jira/browse/HDDS-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3632:
Labels: Triaged newbie (was: newbie)
> HddsDatanodeService cannot be started if HDFS datanode running in same
> machine with same user.
> --
>
> Key: HDDS-3632
> URL: https://issues.apache.org/jira/browse/HDDS-3632
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.5.0
>Reporter: Uma Maheswara Rao G
>Priority: Minor
> Labels: Triaged, newbie
>
> since the service names are same and they both referring to same location for
> pid files, we can not start both services at once.
> Tweak is to export HADOOP_PID_DIR to different location after starting one
> service and start other one.
> It would be better to have different pid file names.
>
>
> {noformat}
> Umas-MacBook-Pro ozone-0.5.0-beta % bin/ozone --daemon start datanode
> datanode is running as process 25167. Stop it first.
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3286) BasicOzoneFileSystem support batchDelete
[ https://issues.apache.org/jira/browse/HDDS-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3286: Target Version/s: 0.7.0 (was: 0.6.0) > BasicOzoneFileSystem support batchDelete > - > > Key: HDDS-3286 > URL: https://issues.apache.org/jira/browse/HDDS-3286 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Filesystem >Reporter: mingchao zhao >Assignee: mingchao zhao >Priority: Major > Labels: Triaged, pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently delete file is to get all the keys in the directory, and then > delete one by one. And the same with rename. This makes for poor performance. > By tested the deletion path with 100,000 files, which took 3718.70 sec. > And rename it took 7327.936. > HDDS-2939 can able to optimize this part, but at present the hdds-2939 is > slow and still a long way to go. So we optimized the batch operation based on > the current interface. We were able to get better performance with this PR > before the hdds-2939 came in. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3632) HddsDatanodeService cannot be started if HDFS datanode running in same machine with same user.
[
https://issues.apache.org/jira/browse/HDDS-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3632:
Priority: Major (was: Minor)
> HddsDatanodeService cannot be started if HDFS datanode running in same
> machine with same user.
> --
>
> Key: HDDS-3632
> URL: https://issues.apache.org/jira/browse/HDDS-3632
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.5.0
>Reporter: Uma Maheswara Rao G
>Priority: Major
> Labels: Triaged, newbie
>
> since the service names are same and they both referring to same location for
> pid files, we can not start both services at once.
> Tweak is to export HADOOP_PID_DIR to different location after starting one
> service and start other one.
> It would be better to have different pid file names.
>
>
> {noformat}
> Umas-MacBook-Pro ozone-0.5.0-beta % bin/ozone --daemon start datanode
> datanode is running as process 25167. Stop it first.
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3619) OzoneManager fails with IllegalArgumentException for cmdType RenameKey
[
https://issues.apache.org/jira/browse/HDDS-3619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3619:
Target Version/s: 0.6.0
Labels: Triaged (was: )
> OzoneManager fails with IllegalArgumentException for cmdType RenameKey
> --
>
> Key: HDDS-3619
> URL: https://issues.apache.org/jira/browse/HDDS-3619
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: HA, Ozone Manager
>Reporter: Lokesh Jain
>Priority: Major
> Labels: Triaged
>
> All Ozone Manager instances on startup fail with IllegalArgumentException for
> command type RenameKey.
> {code:java}
> 2020-05-19 01:26:32,406 WARN
> org.apache.ratis.grpc.server.GrpcServerProtocolService: om1: installSnapshot
> onError, lastRequest: om2->om1#4-t34, previous=(t:34, i:44118),
> leaderCommit=44118, initializing? false, entries: size=1, first=(t:34,
> i:44119), METADATAENTRY(c:44118):
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED:
> cancelled before receiving half close
> 2020-05-19 01:26:33,521 ERROR
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine: Terminating with
> exit status 1: Request cmdType: RenameKey
> traceID: ""
> clientId: "client-E7949F1158CC"
> userInfo {
> userName: "h...@halxg.cloudera.com"
> remoteAddress: "10.17.200.43"
> hostName: "vb0933.halxg.cloudera.com"
> }
> renameKeyRequest {
> keyArgs {
> volumeName: "vol1"
> bucketName: "bucket1"
> keyName: "teragen/100G-terasort-input/"
> dataSize: 0
> modificationTime: 1589872757030
> }
> toKeyName: "user/ljain/.Trash/Current/teragen/100G-terasort-input/"
> }
> failed with exception
> java.lang.IllegalArgumentException: Trying to set updateID to 35984 which is
> not greater than the current value of 42661 for OMKeyInfo{volume='vol1',
> bucket='bucket1', key='teragen/100G-terasort-input/', dataSize='0',
> creationTime='1589876037688', type='RATIS', factor='ONE'}
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)
> at
> org.apache.hadoop.ozone.om.helpers.WithObjectID.setUpdateID(WithObjectID.java:107)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyRenameRequest.validateAndUpdateCache(OMKeyRenameRequest.java:213)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:226)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:428)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:242)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-05-19 01:26:33,526 INFO org.apache.hadoop.ozone.om.OzoneManagerStarter:
> SHUTDOWN_MSG:
> /
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3612) Allow mounting bucket under other volume
[ https://issues.apache.org/jira/browse/HDDS-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3612: Labels: Triaged (was: ) > Allow mounting bucket under other volume > > > Key: HDDS-3612 > URL: https://issues.apache.org/jira/browse/HDDS-3612 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Manager >Reporter: Attila Doroszlai >Assignee: Attila Doroszlai >Priority: Major > Labels: Triaged > > Step 2 from S3 [volume mapping design > doc|https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/docs/content/design/ozone-volume-management.md#solving-the-mapping-problem-2-4-from-the-problem-listing]: > Implement a bind mount mechanic which makes it possible to mount any > volume/buckets to the specific "s3" volume. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3610) Test Recon works with MySQL and Postgres
[ https://issues.apache.org/jira/browse/HDDS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3610: Target Version/s: 0.7.0 > Test Recon works with MySQL and Postgres > > > Key: HDDS-3610 > URL: https://issues.apache.org/jira/browse/HDDS-3610 > Project: Hadoop Distributed Data Store > Issue Type: Test > Components: Ozone Recon >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Priority: Major > > Until now, Recon has only been tested with embedded DBs - Sqlite and Derby, > with Derby being the default. > In theory it should work with MySQL and Postgres without any changes, but we > would like to verify that. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3590) Recon UI: Add visualization for file size distribution
[ https://issues.apache.org/jira/browse/HDDS-3590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3590: Labels: TriagePending (was: ) > Recon UI: Add visualization for file size distribution > --- > > Key: HDDS-3590 > URL: https://issues.apache.org/jira/browse/HDDS-3590 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: Ozone Recon >Affects Versions: 0.5.0 >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: TriagePending > Attachments: Recon UI_ Visualizing file size distribution.pdf > > > Recon has an API endpoint to get file size distribution in Ozone. Add > visualization in Recon UI for this using histograms. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3594) ManagedChannels are leaked in XceiverClientGrpc manager
[
https://issues.apache.org/jira/browse/HDDS-3594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3594:
Target Version/s: 0.6.0
Labels: TriagePending (was: )
> ManagedChannels are leaked in XceiverClientGrpc manager
> ---
>
> Key: HDDS-3594
> URL: https://issues.apache.org/jira/browse/HDDS-3594
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.6.0
>Reporter: Rakesh Radhakrishnan
>Priority: Major
> Labels: TriagePending
>
> XceiverClientGrpc#ManagedChannels are leaked when running {{Hadoop Synthetic
> Load Generator}} pointing to OzoneFS.
> *Stacktrace:*
> {code:java}
> SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=99, target=10.17.248.31:9859}
> was not shutdown properly!!! ~*~*~*
> Make sure to call shutdown()/shutdownNow() and wait until
> awaitTermination() returns true.
> java.lang.RuntimeException: ManagedChannel allocation site
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:94)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:52)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:43)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:518)
> at
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:191)
> at
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.connect(XceiverClientGrpc.java:140)
> at
> org.apache.hadoop.hdds.scm.XceiverClientManager$2.call(XceiverClientManager.java:244)
> at
> org.apache.hadoop.hdds.scm.XceiverClientManager$2.call(XceiverClientManager.java:228)
> at
> org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4876)
> at
> org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
> at
> org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
> at
> org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
> at
> org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
> at
> org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
> at
> org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4871)
> at
> org.apache.hadoop.hdds.scm.XceiverClientManager.getClient(XceiverClientManager.java:228)
> at
> org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:174)
> at
> org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClientForReadData(XceiverClientManager.java:164)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:184)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:133)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:254)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:199)
> at
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:63)
> at java.io.DataInputStream.read(DataInputStream.java:100)
> at
> org.apache.hadoop.fs.loadGenerator.LoadGenerator$DFSClientThread.read(LoadGenerator.java:284)
> at
> org.apache.hadoop.fs.loadGenerator.LoadGenerator$DFSClientThread.nextOp(LoadGenerator.java:268)
> at
> org.apache.hadoop.fs.loadGenerator.LoadGenerator$DFSClientThread.run(LoadGenerator.java:235)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3600) ManagedChannels leaked on ratis pipeline when there are many connection retries
[
https://issues.apache.org/jira/browse/HDDS-3600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3600:
Target Version/s: 0.6.0
Labels: TriagePending (was: )
> ManagedChannels leaked on ratis pipeline when there are many connection
> retries
> ---
>
> Key: HDDS-3600
> URL: https://issues.apache.org/jira/browse/HDDS-3600
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.6.0
>Reporter: Rakesh Radhakrishnan
>Priority: Major
> Labels: TriagePending
> Attachments: HeapHistogram-Snapshot-ManagedChannel-Leaked-001.png,
> outloggenerator-ozonefs-003.log
>
>
> ManagedChannels leaked on ratis pipeline when there are many connection
> retries
> Observed that too many ManagedChannels opened while running Synthetic Hadoop
> load generator.
> Ran benchmark with only one pipeline in the cluster and also ran with only
> two pipelines in the cluster.
> Both the run failed with too many open files and could see many open TCP
> connections for long time and suspecting channel leaks..
> More details below:
> *1)* Execute NNloadGenerator
> {code:java}
> [rakeshr@ve1320 loadOutput]$ ps -ef | grep load
> hdfs 362822 1 19 05:24 pts/000:03:16
> /usr/java/jdk1.8.0_232-cloudera/bin/java -Dproc_jar -Xmx825955249
> -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true
> -Dyarn.log.dir=/var/log/hadoop-yarn -Dyarn.log.file=hadoop.log
> -Dyarn.home.dir=/opt/cloudera/parcels/CDH-7.2.0-1.cdh7.2.0.p0.2982244/lib/hadoop/libexec/../../hadoop-yarn
> -Dyarn.root.logger=INFO,console
> -Djava.library.path=/opt/cloudera/parcels/CDH-7.2.0-1.cdh7.2.0.p0.2982244/lib/hadoop/lib/native
> -Dhadoop.log.dir=/var/log/hadoop-yarn -Dhadoop.log.file=hadoop.log
> -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-7.2.0-1.cdh7.2.0.p0.2982244/lib/hadoop
> -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,console
> -Dhadoop.policy.file=hadoop-policy.xml
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar
> /opt/cloudera/parcels/CDH-7.2.0-1.cdh7.2.0.p0.2982244/jars/hadoop-mapreduce-client-jobclient-3.1.1.7.2.0.0-141-tests.jar
> NNloadGenerator -root o3fs://bucket2.vol2/
> rakeshr 368739 354174 0 05:41 pts/000:00:00 grep --color=auto load
> {code}
> *2)* Active 9858 TCP connections during the run, which is ratis pipeline
> default port.
> {code:java}
> [rakeshr@ve1320 loadOutput]$ sudo lsof -a -p 362822 | grep "9858" | wc
>3229 32290 494080
> [rakeshr@ve1320 loadOutput]$ vi tcp_log
>
> java440633 hdfs 4090u IPv4 271141987 0t0TCP
> ve1320.halxg.cloudera.com:35190->ve1323.halxg.cloudera.com:9858 (ESTABLISHED)
> java440633 hdfs 4091u IPv4 271127918 0t0TCP
> ve1320.halxg.cloudera.com:35192->ve1323.halxg.cloudera.com:9858 (ESTABLISHED)
> java440633 hdfs 4092u IPv4 271038583 0t0TCP
> ve1320.halxg.cloudera.com:59116->ve1323.halxg.cloudera.com:9858 (ESTABLISHED)
> java440633 hdfs 4093u IPv4 271038584 0t0TCP
> ve1320.halxg.cloudera.com:59118->ve1323.halxg.cloudera.com:9858 (ESTABLISHED)
> java440633 hdfs 4095u IPv4 271127920 0t0TCP
> ve1320.halxg.cloudera.com:35196->ve1323.halxg.cloudera.com:9858 (ESTABLISHED)
> [rakeshr@ve1320 loadOutput]$ ^C
> {code}
> *3)* heapdump shows there are 9571 ManagedChanel objects. Heapdump is quite
> large and attached snapshot to this jira.
> *4)* Attached output and threadump of the SyntheticLoadGenerator benchmark
> client process to show the exceptions printed to the console. FYI, this file
> was quite large and have trimmed few repeated exception traces..
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3576) Use direct ratis config in OM Ratis Server
[ https://issues.apache.org/jira/browse/HDDS-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3576: Target Version/s: 0.7.0 > Use direct ratis config in OM Ratis Server > -- > > Key: HDDS-3576 > URL: https://issues.apache.org/jira/browse/HDDS-3576 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: OMHA, Ozone Manager >Reporter: Bharat Viswanadham >Priority: Major > > This Jira is to use getPropertiesByPrefix that match with "ozone.om.ratis" > and set them during ratisServer creation in OzoneManager. > Advantages: > # We can use ratis properties directly, don't need to create corresponding > ozone config. > # When new properties are added in ratis server, we can set them and use > them without any ozone code changes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3582) Update Checkstyle rule
[
https://issues.apache.org/jira/browse/HDDS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3582:
Target Version/s: 0.7.0
> Update Checkstyle rule
> --
>
> Key: HDDS-3582
> URL: https://issues.apache.org/jira/browse/HDDS-3582
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: build
>Affects Versions: 0.6.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
>
> Now. the rules of checkstyle is hadoop style, as we all know, hadoop repo
> started 10 years ago, so some of its rules need to be update.
> For example.
> - 120 length characters of a line make more sense, because most of all have a
> better monitor than 10 years ago.
> - We need more style rules, such as, behind and after "{" and "}" should have
> a space.
> - We should have manage the import into group and order them
> - ModifierOrder is needed
> - Remove the package javadoc rules.
> hope our community getting better and better
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3575) Implement trash clean up on OM
[ https://issues.apache.org/jira/browse/HDDS-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3575: Target Version/s: 0.6.0 Labels: Triaged (was: ) Priority: Critical (was: Major) > Implement trash clean up on OM > -- > > Key: HDDS-3575 > URL: https://issues.apache.org/jira/browse/HDDS-3575 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: om >Reporter: Siyao Meng >Priority: Critical > Labels: Triaged > > We need to implement trash clean up on Ozone Manager. Otherwise files deleted > with Hadoop shell will not be cleaned up. > Inspired from HDDS-3574. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3570) Add Recon UI lint checks and unit tests to Github Actions CI workflow
[ https://issues.apache.org/jira/browse/HDDS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3570: Labels: TriagePending (was: ) > Add Recon UI lint checks and unit tests to Github Actions CI workflow > - > > Key: HDDS-3570 > URL: https://issues.apache.org/jira/browse/HDDS-3570 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: Ozone Recon >Affects Versions: 0.5.0 >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: TriagePending > > Recon UI unit tests and linter checks should be added to Github actions CI. > Optimization: Try to add a workflow in such a way that these checks are run > only if there is a change related to Recon UI files. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3569) Make open keys visible to client
[ https://issues.apache.org/jira/browse/HDDS-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3569: Target Version/s: 0.7.0 > Make open keys visible to client > > > Key: HDDS-3569 > URL: https://issues.apache.org/jira/browse/HDDS-3569 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: om >Affects Versions: 0.5.0 >Reporter: Nanda kumar >Priority: Major > > With support for Filesytem client and Fuse/NFS in Ozone we run into cases > where we are required to make the open file/key visible to client. > This jira is for discussion around this area to check if it is feasible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3568) support change replication number of a exist file
[ https://issues.apache.org/jira/browse/HDDS-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3568: Target Version/s: 0.7.0 > support change replication number of a exist file > - > > Key: HDDS-3568 > URL: https://issues.apache.org/jira/browse/HDDS-3568 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone CLI, Ozone Client, Ozone Datanode, SCM >Reporter: maobaolong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3559) Datanode doesn't handle java heap OutOfMemory exception
[ https://issues.apache.org/jira/browse/HDDS-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3559: Target Version/s: 0.6.0 Labels: Triaged pull-request-available (was: pull-request-available) > Datanode doesn't handle java heap OutOfMemory exception > > > Key: HDDS-3559 > URL: https://issues.apache.org/jira/browse/HDDS-3559 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.5.0 >Reporter: Li Cheng >Priority: Major > Labels: Triaged, pull-request-available > > 2020-05-05 15:47:41,568 [Datanode State Machine Thread - 167] WARN > org.apache.hadoop.ozone.container.common.statemachine.Endpoi > ntStateMachine: Unable to communicate to SCM server at host-10-51-87-181:9861 > for past 0 seconds. > java.io.IOException: com.google.protobuf.ServiceException: > java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.ipc.ProtobufHelper.getRemoteException(ProtobufHelper.java:47) > at > org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolClientSideTranslatorPB.submitRequest(StorageContainerDatanodeProtocolClientSideTranslatorPB.java:118) > at > org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolClientSideTranslatorPB.sendHeartbeat(StorageContainerDatanodeProtocolClientSideTranslatorPB.java:148) > at > org.apache.hadoop.ozone.container.common.states.endpoint.HeartbeatEndpointTask.call(HeartbeatEndpointTask.java:145) > at > org.apache.hadoop.ozone.container.common.states.endpoint.HeartbeatEndpointTask.call(HeartbeatEndpointTask.java:76) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: com.google.protobuf.ServiceException: java.lang.OutOfMemoryError: > Java heap space > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.getReturnMessage(ProtobufRpcEngine.java:293) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:270) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy38.submitRequest(Unknown Source) > at > org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolClientSideTranslatorPB.submitRequest(StorageContainerDatanodeProtocolClientSideTranslatorPB.java:116) > > On a cluster, one datanode stops reporting to SCM while being kept unknown. > The datanode process is still working. Log shows Java heap OOM when it's > serializing protobuf for rpc message. However, datanode silently stops > reports to SCM and the process becomes stale. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] hanishakoneru commented on a change in pull request #986: HDDS-3476. Use persisted transaction info during OM startup in OM StateMachine.
hanishakoneru commented on a change in pull request #986:
URL: https://github.com/apache/hadoop-ozone/pull/986#discussion_r433429260
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java
##
@@ -133,13 +133,6 @@ private OMConfigKeys() {
"ozone.om.ratis.log.purge.gap";
public static final int OZONE_OM_RATIS_LOG_PURGE_GAP_DEFAULT = 100;
- // OM Snapshot configurations
- public static final String OZONE_OM_RATIS_SNAPSHOT_AUTO_TRIGGER_THRESHOLD_KEY
- = "ozone.om.ratis.snapshot.auto.trigger.threshold";
Review comment:
We need this config to be able to control the snapshot trigger from
Ratis side for testing purposes. For some tests (TestOzoneManagerHA, OM HA
robot tests) we need to set a lower value for auto trigger threshold so that
logs can be purged which in turn can instantiate install snapshots.
We can keep it as an internal config and not expose it to users.
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3027,30 +3017,59 @@ public TermIndex installSnapshot(String leaderId) {
DBCheckpoint omDBcheckpoint = getDBCheckpointFromLeader(leaderId);
Path newDBlocation = omDBcheckpoint.getCheckpointLocation();
-// Check if current ratis log index is smaller than the downloaded
-// snapshot index. If yes, proceed by stopping the ratis server so that
-// the OM state can be re-initialized. If no, then do not proceed with
-// installSnapshot.
long lastAppliedIndex = omRatisServer.getLastAppliedTermIndex().getIndex();
-long checkpointSnapshotIndex = omDBcheckpoint.getRatisSnapshotIndex();
-long checkpointSnapshotTermIndex =
-omDBcheckpoint.getRatisSnapshotTerm();
-if (checkpointSnapshotIndex <= lastAppliedIndex) {
- LOG.error("Failed to install checkpoint from OM leader: {}. The last " +
- "applied index: {} is greater than or equal to the checkpoint's " +
- "snapshot index: {}. Deleting the downloaded checkpoint {}",
leaderId,
- lastAppliedIndex, checkpointSnapshotIndex,
- newDBlocation);
- try {
-FileUtils.deleteFully(newDBlocation);
- } catch (IOException e) {
-LOG.error("Failed to fully delete the downloaded DB checkpoint {} " +
-"from OM leader {}.", newDBlocation,
-leaderId, e);
+
+// Check if current ratis log index is smaller than the downloaded
+// checkpoint transaction index. If yes, proceed by stopping the ratis
+// server so that the OM state can be re-initialized. If no, then do not
+// proceed with installSnapshot.
+
+OMTransactionInfo omTransactionInfo = null;
+try {
+ // Set new DB location as DB path
+ OzoneConfiguration tempConfig = getConfiguration();
+
+ Path dbDir = newDBlocation.getParent();
+ if (dbDir != null) {
+tempConfig.set(OZONE_OM_DB_DIRS, dbDir.toString());
+ } else {
+LOG.error("Incorrect DB location path {} received from checkpoint.",
+newDBlocation);
+return null;
+ }
+
+ OMMetadataManager tempMetadataMgr =
+ new OmMetadataManagerImpl(configuration);
Review comment:
Instead of instantiating a new MetadataManagerImpl, can we not directly
load the RocksDB and read the transactionInfo?
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3027,30 +3017,59 @@ public TermIndex installSnapshot(String leaderId) {
DBCheckpoint omDBcheckpoint = getDBCheckpointFromLeader(leaderId);
Path newDBlocation = omDBcheckpoint.getCheckpointLocation();
-// Check if current ratis log index is smaller than the downloaded
-// snapshot index. If yes, proceed by stopping the ratis server so that
-// the OM state can be re-initialized. If no, then do not proceed with
-// installSnapshot.
long lastAppliedIndex = omRatisServer.getLastAppliedTermIndex().getIndex();
-long checkpointSnapshotIndex = omDBcheckpoint.getRatisSnapshotIndex();
-long checkpointSnapshotTermIndex =
-omDBcheckpoint.getRatisSnapshotTerm();
-if (checkpointSnapshotIndex <= lastAppliedIndex) {
- LOG.error("Failed to install checkpoint from OM leader: {}. The last " +
- "applied index: {} is greater than or equal to the checkpoint's " +
- "snapshot index: {}. Deleting the downloaded checkpoint {}",
leaderId,
- lastAppliedIndex, checkpointSnapshotIndex,
- newDBlocation);
- try {
-FileUtils.deleteFully(newDBlocation);
- } catch (IOException e) {
-LOG.error("Failed to fully delete the downloaded DB checkpoint {} " +
-"from OM leader {}.", newDBlocation,
-leaderId, e);
+
+// Check if current ratis log index is smaller than the downloaded
+// checkpoint transaction index. If yes, proceed by stopping the
[jira] [Updated] (HDDS-3567) Make replicationfactor can be configurable to any number
[ https://issues.apache.org/jira/browse/HDDS-3567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3567: Target Version/s: 0.7.0 > Make replicationfactor can be configurable to any number > > > Key: HDDS-3567 > URL: https://issues.apache.org/jira/browse/HDDS-3567 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: om, Ozone CLI, Ozone Datanode, Ozone Manager, SCM >Affects Versions: 0.6.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > > As there is a feature request that Ozone should support any replication > number of a file, we have the following subtask to do. > The following is a simple design document. > https://docs.qq.com/doc/DV2N6bWdCcnJVc3Rk -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3562) Datanodes should send ICR when a container replica deletion is successful
[ https://issues.apache.org/jira/browse/HDDS-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3562: Labels: Triaged pull-request-available (was: pull-request-available) > Datanodes should send ICR when a container replica deletion is successful > - > > Key: HDDS-3562 > URL: https://issues.apache.org/jira/browse/HDDS-3562 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.5.0 >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Major > Labels: Triaged, pull-request-available > > Whenever a datanode executes the delete container command and deletes the > container replica, it has to immediately send an ICR to update the container > replica state in SCM. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3545) MR Jobhistory cannot work well with o3fs hadoop compatible file system
[ https://issues.apache.org/jira/browse/HDDS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3545: Target Version/s: 0.6.0 Labels: TriagePending (was: ) > MR Jobhistory cannot work well with o3fs hadoop compatible file system > -- > > Key: HDDS-3545 > URL: https://issues.apache.org/jira/browse/HDDS-3545 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Reporter: maobaolong >Priority: Major > Labels: TriagePending > > After take a look at the code of JobHistory, i see jobhistory use > `fs.getModificationTime()` to get the directory modification time, and use it > as a condition of starting scan the job directories. > But, for ozone, wile insert a child to a `directory`, the modification time > of the `directory` don't update now. > So we should update the modification time of `directory`, otherwise, MR > Jobhistory and some other component which use the modification time of > `directory` cannot work as expected. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3546) The default config items of ScmClientConfig don't appear in the ozone-filesystem-xxx-jar
[
https://issues.apache.org/jira/browse/HDDS-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3546:
Target Version/s: 0.6.0
Labels: Triaged pull-request-available (was:
pull-request-available)
> The default config items of ScmClientConfig don't appear in the
> ozone-filesystem-xxx-jar
>
>
> Key: HDDS-3546
> URL: https://issues.apache.org/jira/browse/HDDS-3546
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Filesystem, SCM Client
>Affects Versions: 0.6.0
>Reporter: maobaolong
>Assignee: Simon Su
>Priority: Major
> Labels: Triaged, pull-request-available
>
> for example `scm.container.client.idle.threshold` should in the default
> config file of ozone, Without it, the following logic won't work well while
> put a key.
> {code:java}
> long staleThresholdMs = clientConf.getStaleThreshold(MILLISECONDS);
> this.clientCache = CacheBuilder.newBuilder()
> .expireAfterAccess(staleThresholdMs, MILLISECONDS)
> .maximumSize(clientConf.getMaxSize())
> {code}
> because the staleThresholdMs is 0, if we don't config it, it should return
> the default value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3518) Add a freon generator to create directory tree and create files in each directory
[
https://issues.apache.org/jira/browse/HDDS-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3518:
Target Version/s: 0.6.0
Labels: Triaged pull-request-available (was:
pull-request-available)
> Add a freon generator to create directory tree and create files in each
> directory
> -
>
> Key: HDDS-3518
> URL: https://issues.apache.org/jira/browse/HDDS-3518
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Tools
>Reporter: Rakesh Radhakrishnan
>Assignee: Rakesh Radhakrishnan
>Priority: Major
> Labels: Triaged, pull-request-available
>
> This Jira proposes to add a functionality to freon tool to create directories
> in tree structure.
> *Proposed structure:*
> {code:java}
> /*
> Nested directories will be created like this,
> suppose you pass depth=3, span=3 and number of tests=2
> Directory Structure:-
> |-- Dir111
> |
> |-- Dir11 --|-- Dir112
> | |
> | |-- Dir113
> |
> |
> | |-- Dir121
> | |
>Dir1 --|-- Dir12 --|-- Dir122
> | |
> | |-- Dir123
> |
> |
> | |-- Dir131
> | |
> |-- Dir13 --|-- Dir132
> |
> |-- Dir133
> In each directory 'c' number of files to be written in each directory
> with file size 'g' will be created.
>*/
> {code}
> This is basically useful to determine the performance evaluation of ozone,
> when creating defined number of nested directories and also create given
> number of files into each directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3509) Closing container with unhealthy replica on open pipeline
[ https://issues.apache.org/jira/browse/HDDS-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3509: Labels: TriagePending (was: ) > Closing container with unhealthy replica on open pipeline > - > > Key: HDDS-3509 > URL: https://issues.apache.org/jira/browse/HDDS-3509 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.5.0 >Reporter: Nilotpal Nandi >Assignee: Nanda kumar >Priority: Major > Labels: TriagePending > > When a container replica of an OPEN container is marked as UNHEALTHY, SCM > tries to close the container. > If the pipeline is still healthy, we try to close the container via Ratis. > We could run into a scenario where the datanode which marked the container > replica as UNHEALTHY is the pipeline leader. In such case that datanode > (leader) should process the close container command even though the container > replica is in UNHEALTHY state. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3498) Address already in use Should shutdown the datanode with FATAL log and point out the port and configure key
[
https://issues.apache.org/jira/browse/HDDS-3498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3498:
Target Version/s: 0.6.0
Labels: Triaged (was: )
> Address already in use Should shutdown the datanode with FATAL log and point
> out the port and configure key
> ---
>
> Key: HDDS-3498
> URL: https://issues.apache.org/jira/browse/HDDS-3498
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.6.0
>Reporter: maobaolong
>Priority: Minor
> Labels: Triaged
>
> Now, the datanode process cannot work because the port is in use, but the
> process still live.
> Further more, i guess the in use port is 9861, but it isn't, after look the
> source code, i find it is the `dfs.container.ipc`, default port is 9859, this
> port should appear with the following exception. I think this error should be
> in FATAL level, and we can terminate the datanode process.
> {code:java}
> 2020-04-21 15:53:05,436 [Datanode State Machine Thread - 0] WARN
> org.apache.hadoop.ozone.container.common.statemachine.EndpointStateMachine:
> Unable to communicate to SCM server at 127.0.0.1:9861 for past 300 seconds.
> java.io.IOException: Failed to bind
> at
> org.apache.ratis.thirdparty.io.grpc.netty.NettyServer.start(NettyServer.java:246)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:184)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:90)
> at
> org.apache.hadoop.ozone.container.common.transport.server.XceiverServerGrpc.start(XceiverServerGrpc.java:141)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:235)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:113)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:42)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at
> org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:132)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:551)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1345)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:503)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:488)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:984)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:247)
> at
> org.apache.ratis.thirdparty.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:355)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416)
> at
> org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:515)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
> at
> org.apache.ratis.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> ... 1 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HDDS-3497) Host cannot access DN in the docker container when using docker-compose
[ https://issues.apache.org/jira/browse/HDDS-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3497: Target Version/s: 0.7.0 > Host cannot access DN in the docker container when using docker-compose > --- > > Key: HDDS-3497 > URL: https://issues.apache.org/jira/browse/HDDS-3497 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: docker >Affects Versions: 0.6.0 >Reporter: maobaolong >Priority: Major > Attachments: screenshot-1.png > > > Now i using docker-compose to start a ozone cluster, and i create volume and > bucket, but i cannot put some file into this ozone cluster in my host node. > Following is the step to reproduce. > ozone-0.6.0-SNAPSHOT/compose/ozone/run.sh > pwd=ozone-0.6.0-SNAPSHOT > bin/ozone sh vol create myvol > bin/ozone sh bucket create myvol/mybucket > bin/ozone sh key put /myvol/mybucket/README.md README.md > The last command stuck. > And i do some debug work. I see the ratis pipeline here with nodes using > ip:port, in the host node, i cannot access these ip in another network > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3505) Cleanup remaining S3 methods
[
https://issues.apache.org/jira/browse/HDDS-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3505:
Labels: TriagePending (was: )
> Cleanup remaining S3 methods
>
>
> Key: HDDS-3505
> URL: https://issues.apache.org/jira/browse/HDDS-3505
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Manager
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: TriagePending
>
> Investigate moving {{createS3Bucket}}, {{getS3Bucket}} and {{deleteS3Bucket}}
> from {{ObjectStore}} to S3 Gateway.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3486) Recon cannot track missing containers that were created and went missing while it is down.
[ https://issues.apache.org/jira/browse/HDDS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3486: Fix Version/s: (was: 0.6.0) Target Version/s: 0.6.0 Labels: TriagePending (was: ) > Recon cannot track missing containers that were created and went missing > while it is down. > -- > > Key: HDDS-3486 > URL: https://issues.apache.org/jira/browse/HDDS-3486 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Recon >Affects Versions: 0.6.0 >Reporter: Aravindan Vijayan >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: TriagePending > > Since Recon relies on Container reports from DNs to track replicas, if a new > container was created and went missing when Recon was down, it will not be > able to tag it as a missing container. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3485) Use typesafe ContainerID instead of long value
[
https://issues.apache.org/jira/browse/HDDS-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3485:
Labels: TriagePending (was: )
> Use typesafe ContainerID instead of long value
> --
>
> Key: HDDS-3485
> URL: https://issues.apache.org/jira/browse/HDDS-3485
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.5.0
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Labels: TriagePending
>
> We should only use typesafe {{ContainerID}} reference in all the places
> inside SCM and avoid using long value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3478) Add a scheduler based flush APIn in KeyOutputStream
[ https://issues.apache.org/jira/browse/HDDS-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3478: Labels: TriagePending (was: ) > Add a scheduler based flush APIn in KeyOutputStream > --- > > Key: HDDS-3478 > URL: https://issues.apache.org/jira/browse/HDDS-3478 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.6.0 >Reporter: Shashikant Banerjee >Assignee: mingchao zhao >Priority: Major > Labels: TriagePending > Fix For: 0.6.0 > > > [https://github.com/apache/hadoop-ozone/pull/716] proposes to add an > scheduler based flush implementation to mimic S3AFlush(). The Jira aims to > address that. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on pull request #941: HDDS-3574. Implement ofs://: Override getTrashRoot
xiaoyuyao commented on pull request #941: URL: https://github.com/apache/hadoop-ozone/pull/941#issuecomment-637135141 +1, feel free to merge if there is no additional comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3466) Improve filterViableNodes performance in pipeline creation
[
https://issues.apache.org/jira/browse/HDDS-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3466:
Description:
Per [~sodonnell]'s investigation, pipeline creation may have performance issue
once the load-sorting algorithm in HDDS-3139.
This task is to track potential performance bottleneck caused by this sorting
operation for pipeline creation in large scale cluster.
I am a little concerned about the expense of forming the list of healthy nodes
on large clusters. We have to do quite a lot of work to form a list and then
only use 3 nodes from the list. Even the method {{currentPipelineCount()}}
needs to do a few map lookups per node to get the current pipeline count. This
is the case even before this change. Creating a pipeline on a large cluster
would be expensive already, but this change probably makes it worse, due to the
sort needed. I know it was me who suggested the sort.
I think the code as it is will work OK upto about 1000 nodes, and then the
performance will drop off as the number of nodes goes toward 10k.
Eg here are some benchmarks I created using this test code, which is similar to
what we are doing in {{filterViableNodes()}}:
{{ public List sortingWithMap(BenchmarkState state) \{
return state.otherList.stream()
.map(o -> new Mock(o, state.rand.nextInt(20)))
.filter(o -> o.getSize() <= 20)
.sorted(Comparator.comparingInt(Mock::getSize))
.map(o -> o.getObject())
.collect(Collectors.toList());
}}}
The OPs per second for various list sizes are:
{{Benchmark (listSize) Mode Cnt Score Error Units
Sorting.sortingWithMap 100 thrpt3 113948.345 ± 446.426 ops/s
Sorting.sortingWithMap1000 thrpt39468.507 ± 894.138 ops/s
Sorting.sortingWithMap5000 thrpt31931.612 ± 263.919 ops/s
Sorting.sortingWithMap 1 thrpt3 970.745 ± 25.823 ops/s
Sorting.sortingWithMap 10 thrpt3 87.684 ± 35.438 ops/s}}
For a 1000 node cluster, with 10 pipelines per node, we would be looking at
about 1 second to form all the piplines.
For a 5k node cluster, it would be about 25 seconds
For a 10k node cluster it would be 103 seconds, but even here, that would be at
close to 1000 pipelines per second.
was:
Per [~sodonnell]'s investigation, pipeline creation may have performance issue
once the load-sorting algorithm in
https://issues.apache.org/jira/browse/HDDS-3139.
This task is to track potential performance bottleneck caused by this sorting
operation for pipeline creation in large scale cluster.
I am a little concerned about the expense of forming the list of healthy nodes
on large clusters. We have to do quite a lot of work to form a list and then
only use 3 nodes from the list. Even the method {{currentPipelineCount()}}
needs to do a few map lookups per node to get the current pipeline count. This
is the case even before this change. Creating a pipeline on a large cluster
would be expensive already, but this change probably makes it worse, due to the
sort needed. I know it was me who suggested the sort.
I think the code as it is will work OK upto about 1000 nodes, and then the
performance will drop off as the number of nodes goes toward 10k.
Eg here are some benchmarks I created using this test code, which is similar to
what we are doing in {{filterViableNodes()}}:
{{ public List sortingWithMap(BenchmarkState state) \{
return state.otherList.stream()
.map(o -> new Mock(o, state.rand.nextInt(20)))
.filter(o -> o.getSize() <= 20)
.sorted(Comparator.comparingInt(Mock::getSize))
.map(o -> o.getObject())
.collect(Collectors.toList());
}}}
The OPs per second for various list sizes are:
{{Benchmark (listSize) Mode Cnt Score Error Units
Sorting.sortingWithMap 100 thrpt3 113948.345 ± 446.426 ops/s
Sorting.sortingWithMap1000 thrpt39468.507 ± 894.138 ops/s
Sorting.sortingWithMap5000 thrpt31931.612 ± 263.919 ops/s
Sorting.sortingWithMap 1 thrpt3 970.745 ± 25.823 ops/s
Sorting.sortingWithMap 10 thrpt3 87.684 ± 35.438 ops/s}}
For a 1000 node cluster, with 10 pipelines per node, we would be looking at
about 1 second to form all the piplines.
For a 5k node cluster, it would be about 25 seconds
For a 10k node cluster it would be 103 seconds, but even here, that would be at
close to 1000 pipelines per second.
> Improve filterViableNodes performance in pipeline creation
> --
>
> Key: HDDS-3466
> URL: https://issues.apache.org/jira/browse/HDDS-3466
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>
[jira] [Updated] (HDDS-3466) Improve filterViableNodes performance in pipeline creation
[
https://issues.apache.org/jira/browse/HDDS-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3466:
Target Version/s: 0.6.0
Labels: TriagePending (was: )
> Improve filterViableNodes performance in pipeline creation
> --
>
> Key: HDDS-3466
> URL: https://issues.apache.org/jira/browse/HDDS-3466
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Affects Versions: 0.5.0
>Reporter: Li Cheng
>Priority: Major
> Labels: TriagePending
>
> Per [~sodonnell]'s investigation, pipeline creation may have performance
> issue once the load-sorting algorithm in HDDS-3139.
>
> This task is to track potential performance bottleneck caused by this sorting
> operation for pipeline creation in large scale cluster.
>
> I am a little concerned about the expense of forming the list of healthy
> nodes on large clusters. We have to do quite a lot of work to form a list and
> then only use 3 nodes from the list. Even the method
> {{currentPipelineCount()}} needs to do a few map lookups per node to get the
> current pipeline count. This is the case even before this change. Creating a
> pipeline on a large cluster would be expensive already, but this change
> probably makes it worse, due to the sort needed. I know it was me who
> suggested the sort.
> I think the code as it is will work OK upto about 1000 nodes, and then the
> performance will drop off as the number of nodes goes toward 10k.
> Eg here are some benchmarks I created using this test code, which is similar
> to what we are doing in {{filterViableNodes()}}:
>
> {{ public List sortingWithMap(BenchmarkState state) \{
> return state.otherList.stream()
> .map(o -> new Mock(o, state.rand.nextInt(20)))
> .filter(o -> o.getSize() <= 20)
> .sorted(Comparator.comparingInt(Mock::getSize))
> .map(o -> o.getObject())
> .collect(Collectors.toList());
> }}}
> The OPs per second for various list sizes are:
>
> {{Benchmark (listSize) Mode Cnt Score Error Units
> Sorting.sortingWithMap 100 thrpt3 113948.345 ± 446.426 ops/s
> Sorting.sortingWithMap1000 thrpt39468.507 ± 894.138 ops/s
> Sorting.sortingWithMap5000 thrpt31931.612 ± 263.919 ops/s
> Sorting.sortingWithMap 1 thrpt3 970.745 ± 25.823 ops/s
> Sorting.sortingWithMap 10 thrpt3 87.684 ± 35.438 ops/s}}
> For a 1000 node cluster, with 10 pipelines per node, we would be looking at
> about 1 second to form all the piplines.
> For a 5k node cluster, it would be about 25 seconds
> For a 10k node cluster it would be 103 seconds, but even here, that would be
> at close to 1000 pipelines per second.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3477) Disable partial chunk write during flush() call in ozone client by default
[ https://issues.apache.org/jira/browse/HDDS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3477: Labels: Triaged pull-request-available (was: pull-request-available) > Disable partial chunk write during flush() call in ozone client by default > -- > > Key: HDDS-3477 > URL: https://issues.apache.org/jira/browse/HDDS-3477 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Reporter: Shashikant Banerjee >Assignee: mingchao zhao >Priority: Major > Labels: Triaged, pull-request-available > Fix For: 0.6.0 > > > Currently, Ozone client flushes the partial chunks as well during flush() > call by default. > [https://github.com/apache/hadoop-ozone/pull/716] proposes to add a > configuration to disallow partial chunk flush during flush() call. This Jira > aims to enable the config on by default to mimic the default hdfs flush() > behaviour and fix any failing unit tests associated with the change. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3414) Multi-raft does not create two distinct pipelines for 4 DataNodes
[
https://issues.apache.org/jira/browse/HDDS-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17121360#comment-17121360
]
Arpit Agarwal commented on HDDS-3414:
-
This looks like a nice find [~smajeti]. FYI [~nanda] [~timmylicheng]
> Multi-raft does not create two distinct pipelines for 4 DataNodes
> -
>
> Key: HDDS-3414
> URL: https://issues.apache.org/jira/browse/HDDS-3414
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.5.0
>Reporter: Srinivasu Majeti
>Priority: Major
> Labels: Triaged
>
> I have got 4 DNs with ozone.datanode.pipeline.limit set to 3
> I see two pipelines created with *Factor:THREE* but both pipelines have the
> same set of nodes (node2,node3,node4) and do not include node1 at all :( .
>
> [root@c1265-node1 data]# ozone admin pipeline list
> Pipeline[ Id: 46723ec2-cc7f-422a-bd06-d28963919b9a, Nodes:
> 0ed0e13b-0ab0-48e2-ad11-1fca721d3095\{ip: 172.25.39.138, host:
> c1265-node2.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN,
> leaderId:0ed0e13b-0ab0-48e2-ad11-1fca721d3095,
> CreationTimestamp2020-04-16T14:56:55.675Z]
> Pipeline[ Id: 5379c9ef-3c99-481f-980d-0f821ab320dd, Nodes:
> ab5b82f8-836f-4e74-a3eb-9adec7a0f398\{ip: 172.25.33.148, host:
> c1265-node3.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN,
> leaderId:ab5b82f8-836f-4e74-a3eb-9adec7a0f398,
> CreationTimestamp2020-04-16T14:56:55.676Z]
> Pipeline[ Id: 8588424d-01fa-4a91-8462-4510bce43c1c, Nodes:
> 987ef87a-ea9f-453f-9bd7-39a27c7e4f7c\{ip: 172.25.38.32, host:
> c1265-node4.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN,
> leaderId:987ef87a-ea9f-453f-9bd7-39a27c7e4f7c,
> CreationTimestamp2020-04-16T14:56:55.678Z]
> Pipeline[ Id: 33cb3301-9753-45f8-aac8-d86412997170, Nodes:
> 0ed0e13b-0ab0-48e2-ad11-1fca721d3095\{ip: 172.25.39.138, host:
> c1265-node2.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}987ef87a-ea9f-453f-9bd7-39a27c7e4f7c\{ip: 172.25.38.32,
> host: c1265-node4.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}ab5b82f8-836f-4e74-a3eb-9adec7a0f398\{ip: 172.25.33.148,
> host: c1265-node3.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN,
> leaderId:ab5b82f8-836f-4e74-a3eb-9adec7a0f398,
> CreationTimestamp2020-04-16T14:57:00.240Z]
> Pipeline[ Id: 2ba697b0-39b5-4be6-8c53-b65c74542f48, Nodes:
> 84ac4ab6-8f09-4e6e-9759-8863060a685e\{ip: 172.25.37.142, host:
> c1265-node1.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN,
> leaderId:84ac4ab6-8f09-4e6e-9759-8863060a685e,
> CreationTimestamp2020-04-16T14:56:55.672Z]
> Pipeline[ Id: 667dd634-21d2-49cf-977b-cd95acf031fc, Nodes:
> 0ed0e13b-0ab0-48e2-ad11-1fca721d3095\{ip: 172.25.39.138, host:
> c1265-node2.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}ab5b82f8-836f-4e74-a3eb-9adec7a0f398\{ip: 172.25.33.148,
> host: c1265-node3.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}987ef87a-ea9f-453f-9bd7-39a27c7e4f7c\{ip: 172.25.38.32,
> host: c1265-node4.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN,
> leaderId:ab5b82f8-836f-4e74-a3eb-9adec7a0f398,
> CreationTimestamp2020-04-16T14:56:55.677Z]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3452) Introduce priority queue for SCMCommands in datanode
[
https://issues.apache.org/jira/browse/HDDS-3452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3452:
Target Version/s: 0.7.0 (was: 0.6.0)
> Introduce priority queue for SCMCommands in datanode
>
>
> Key: HDDS-3452
> URL: https://issues.apache.org/jira/browse/HDDS-3452
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
>Affects Versions: 0.5.0
>Reporter: Nanda kumar
>Priority: Major
>
> Datanode maintains a command queue that is used for processing all the
> commands sent by SCM ({{StateContext#commandQueue}}).
> We should make this a priority queue and add priority to the commands sent by
> SCM.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3413) Ozone documentation to be revised for OM HA Support
[ https://issues.apache.org/jira/browse/HDDS-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3413: Target Version/s: 0.6.0 Labels: TriagePending (was: ) > Ozone documentation to be revised for OM HA Support > --- > > Key: HDDS-3413 > URL: https://issues.apache.org/jira/browse/HDDS-3413 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: documentation >Affects Versions: 0.5.0 >Reporter: Srinivasu Majeti >Priority: Major > Labels: TriagePending > > As OM HA is now supported in current version , We might need to update all > documentation pages wherever service id is applicable and any new parameters > that we might need to configure in core-site.xml for service id access for > remote clusters etc. And all volume/bucket/key CLI syntaxes including service > id for OM HA enabled clusters should be included in documentation -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3414) Multi-raft does not create two distinct pipelines for 4 DataNodes
[
https://issues.apache.org/jira/browse/HDDS-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3414:
Target Version/s: 0.6.0
Labels: Triaged (was: )
> Multi-raft does not create two distinct pipelines for 4 DataNodes
> -
>
> Key: HDDS-3414
> URL: https://issues.apache.org/jira/browse/HDDS-3414
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.5.0
>Reporter: Srinivasu Majeti
>Priority: Major
> Labels: Triaged
>
> I have got 4 DNs with ozone.datanode.pipeline.limit set to 3
> I see two pipelines created with *Factor:THREE* but both pipelines have the
> same set of nodes (node2,node3,node4) and do not include node1 at all :( .
>
> [root@c1265-node1 data]# ozone admin pipeline list
> Pipeline[ Id: 46723ec2-cc7f-422a-bd06-d28963919b9a, Nodes:
> 0ed0e13b-0ab0-48e2-ad11-1fca721d3095\{ip: 172.25.39.138, host:
> c1265-node2.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN,
> leaderId:0ed0e13b-0ab0-48e2-ad11-1fca721d3095,
> CreationTimestamp2020-04-16T14:56:55.675Z]
> Pipeline[ Id: 5379c9ef-3c99-481f-980d-0f821ab320dd, Nodes:
> ab5b82f8-836f-4e74-a3eb-9adec7a0f398\{ip: 172.25.33.148, host:
> c1265-node3.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN,
> leaderId:ab5b82f8-836f-4e74-a3eb-9adec7a0f398,
> CreationTimestamp2020-04-16T14:56:55.676Z]
> Pipeline[ Id: 8588424d-01fa-4a91-8462-4510bce43c1c, Nodes:
> 987ef87a-ea9f-453f-9bd7-39a27c7e4f7c\{ip: 172.25.38.32, host:
> c1265-node4.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN,
> leaderId:987ef87a-ea9f-453f-9bd7-39a27c7e4f7c,
> CreationTimestamp2020-04-16T14:56:55.678Z]
> Pipeline[ Id: 33cb3301-9753-45f8-aac8-d86412997170, Nodes:
> 0ed0e13b-0ab0-48e2-ad11-1fca721d3095\{ip: 172.25.39.138, host:
> c1265-node2.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}987ef87a-ea9f-453f-9bd7-39a27c7e4f7c\{ip: 172.25.38.32,
> host: c1265-node4.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}ab5b82f8-836f-4e74-a3eb-9adec7a0f398\{ip: 172.25.33.148,
> host: c1265-node3.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN,
> leaderId:ab5b82f8-836f-4e74-a3eb-9adec7a0f398,
> CreationTimestamp2020-04-16T14:57:00.240Z]
> Pipeline[ Id: 2ba697b0-39b5-4be6-8c53-b65c74542f48, Nodes:
> 84ac4ab6-8f09-4e6e-9759-8863060a685e\{ip: 172.25.37.142, host:
> c1265-node1.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN,
> leaderId:84ac4ab6-8f09-4e6e-9759-8863060a685e,
> CreationTimestamp2020-04-16T14:56:55.672Z]
> Pipeline[ Id: 667dd634-21d2-49cf-977b-cd95acf031fc, Nodes:
> 0ed0e13b-0ab0-48e2-ad11-1fca721d3095\{ip: 172.25.39.138, host:
> c1265-node2.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}ab5b82f8-836f-4e74-a3eb-9adec7a0f398\{ip: 172.25.33.148,
> host: c1265-node3.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}987ef87a-ea9f-453f-9bd7-39a27c7e4f7c\{ip: 172.25.38.32,
> host: c1265-node4.supportlab.cloudera.com, networkLocation: /default-rack,
> certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN,
> leaderId:ab5b82f8-836f-4e74-a3eb-9adec7a0f398,
> CreationTimestamp2020-04-16T14:56:55.677Z]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3412) Ozone shell should have commands to delete non-empty volumes / buckets recursively
[ https://issues.apache.org/jira/browse/HDDS-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3412: Target Version/s: 0.7.0 > Ozone shell should have commands to delete non-empty volumes / buckets > recursively > --- > > Key: HDDS-3412 > URL: https://issues.apache.org/jira/browse/HDDS-3412 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone CLI >Affects Versions: 0.4.0 >Reporter: Gaurav Sharma >Assignee: Sadanand Shenoy >Priority: Major > > Currently we cannot delete non-empty volumes or buckets or even multiple keys > recursively from ozone shell. A small utility or additional commands in ozone > shell / CLI would be great help. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on pull request #949: HDDS-3621. Separate client/server/admin proto files of HDDS to separated subprojects
xiaoyuyao commented on pull request #949: URL: https://github.com/apache/hadoop-ozone/pull/949#issuecomment-637133168 bq. The idea was to put ALL the intra-server related to one service / proto file of each of the servers. If something is used between two servers it should be part of a dedicated service (but only one service, we don't need separated block / container location services if they are used in the same way) That sounds good to me. The original reason of separating block and container is for cBlock which only needs container service but not block service. bq. I can do it (as the project name is already a classifier) but I think it's less confusing to use exactly the same name everwhere. Good point. Let's keep the same as-is. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3395) Move the protobuf convert code to the OMHelper
[ https://issues.apache.org/jira/browse/HDDS-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3395: Target Version/s: 0.6.0 Labels: Triaged pull-request-available (was: pull-request-available) > Move the protobuf convert code to the OMHelper > -- > > Key: HDDS-3395 > URL: https://issues.apache.org/jira/browse/HDDS-3395 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: om >Affects Versions: 0.6.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Minor > Labels: Triaged, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3379) Clients unable to failover after the OzoneManager leader is restart in MiniOzoneChaosCluster
[
https://issues.apache.org/jira/browse/HDDS-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3379:
Target Version/s: 0.6.0
Labels: MiniOzoneChaosCluster TriagePending (was:
MiniOzoneChaosCluster)
> Clients unable to failover after the OzoneManager leader is restart in
> MiniOzoneChaosCluster
>
>
> Key: HDDS-3379
> URL: https://issues.apache.org/jira/browse/HDDS-3379
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Mukul Kumar Singh
>Priority: Major
> Labels: MiniOzoneChaosCluster, TriagePending
>
> Clients unable to failover after the OzoneManager leader is restart in
> MiniOzoneChaosCluster.
> This happens after the following restart events.
> {code}
> ➜ chaos-2020-04-11-21-51-52-IST egrep "iniOzoneHAClusterImp|Failures"
> complete.log
> 2020-04-11 21:52:08,296
> [org.apache.hadoop.ozone.TestMiniChaosOzoneCluster.main()] INFO
> ozone.MiniOzoneHAClusterImpl
> (MiniOzoneHAClusterImpl.java:createOMService(373)) - Started OzoneManager RPC
> server at localhost/127.0.0.1:10804
> 2020-04-11 21:52:08,387
> [org.apache.hadoop.ozone.TestMiniChaosOzoneCluster.main()] INFO
> ozone.MiniOzoneHAClusterImpl
> (MiniOzoneHAClusterImpl.java:createOMService(373)) - Started OzoneManager RPC
> server at localhost/127.0.0.1:10810
> 2020-04-11 21:52:08,485
> [org.apache.hadoop.ozone.TestMiniChaosOzoneCluster.main()] INFO
> ozone.MiniOzoneHAClusterImpl
> (MiniOzoneHAClusterImpl.java:createOMService(373)) - Started OzoneManager RPC
> server at localhost/127.0.0.1:10816
> 2020-04-11 21:52:22,845
> [org.apache.hadoop.ozone.TestMiniChaosOzoneCluster.main()] INFO
> failure.Failures (FailureManager.java:start(66)) - starting failure manager
> 60 60 SECONDS
> 2020-04-11 21:53:22,850 [pool-59-thread-1] INFO failure.Failures
> (FailureManager.java:fail(56)) - time failure with OzoneManagerRestartFailure
> 2020-04-11 21:53:22,853 [pool-59-thread-1] INFO ozone.MiniOzoneHAClusterImpl
> (MiniOzoneHAClusterImpl.java:shutdownOzoneManager(211)) - Shutting down
> OzoneManager omNode-3
> 2020-04-11 21:53:22,988 [pool-59-thread-1] INFO ozone.MiniOzoneHAClusterImpl
> (MiniOzoneHAClusterImpl.java:restartOzoneManager(228)) - Restarting
> OzoneManager omNode-3
> at
> org.apache.hadoop.ozone.MiniOzoneHAClusterImpl.restartOzoneManager(MiniOzoneHAClusterImpl.java:229)
> at
> org.apache.hadoop.ozone.MiniOzoneHAClusterImpl.restartOzoneManager(MiniOzoneHAClusterImpl.java:223)
> at
> org.apache.hadoop.ozone.failure.Failures$OzoneManagerRestartFailure.lambda$fail$0(Failures.java:101)
> at
> org.apache.hadoop.ozone.failure.Failures$OzoneManagerRestartFailure.fail(Failures.java:98)
> 2020-04-11 21:54:22,849 [pool-59-thread-1] INFO failure.Failures
> (FailureManager.java:fail(56)) - time failure with OzoneManagerRestartFailure
> 2020-04-11 21:54:22,850 [pool-59-thread-1] INFO ozone.MiniOzoneHAClusterImpl
> (MiniOzoneHAClusterImpl.java:shutdownOzoneManager(211)) - Shutting down
> OzoneManager omNode-1
> 2020-04-11 21:54:22,895 [pool-59-thread-1] INFO ozone.MiniOzoneHAClusterImpl
> (MiniOzoneHAClusterImpl.java:restartOzoneManager(228)) - Restarting
> OzoneManager omNode-1
> at
> org.apache.hadoop.ozone.MiniOzoneHAClusterImpl.restartOzoneManager(MiniOzoneHAClusterImpl.java:229)
> at
> org.apache.hadoop.ozone.MiniOzoneHAClusterImpl.restartOzoneManager(MiniOzoneHAClusterImpl.java:223)
> at
> org.apache.hadoop.ozone.failure.Failures$OzoneManagerRestartFailure.lambda$fail$0(Failures.java:101)
> at
> org.apache.hadoop.ozone.failure.Failures$OzoneManagerRestartFailure.fail(Failures.java:98)
> ➜ chaos-2020-04-11-21-51-52-IST
> {code}
> This results in the following exception.
> {code}
> 2020-04-11 21:54:24,201 [pool-360-thread-4] ERROR
> loadgenerators.LoadExecutors (LoadExecutors.java:load(67)) -
> FilesystemLoadGenerator LOADGEN: Exiting due to exception
> java.io.IOException: java.io.IOException: Could not determine or connect to
> OM Leader.
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:229)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:199)
> at
> org.apache.hadoop.fs.ozone.OzoneFSOutputStream.write(OzoneFSOutputStream.java:46)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
> at
> org.apache.hadoop.ozone.utils.LoadBucket$WriteOp.doPostOp(LoadBucket.java:176)
> at
>
[jira] [Updated] (HDDS-3388) Add bucket encryption key info to bucket create audit log
[ https://issues.apache.org/jira/browse/HDDS-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3388: Fix Version/s: (was: 0.6.0) Target Version/s: 0.6.0 Labels: Triaged pull-request-available (was: pull-request-available) > Add bucket encryption key info to bucket create audit log > - > > Key: HDDS-3388 > URL: https://issues.apache.org/jira/browse/HDDS-3388 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager >Affects Versions: 0.5.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Minor > Labels: Triaged, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The current audit log does not include the bucket encryption key information. > This ticket is opened to add that. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3367) Trash Support for Ozone FileSystem
[ https://issues.apache.org/jira/browse/HDDS-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3367: Labels: TriagePending (was: ) > Trash Support for Ozone FileSystem > -- > > Key: HDDS-3367 > URL: https://issues.apache.org/jira/browse/HDDS-3367 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Filesystem >Affects Versions: 0.6.0 >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Minor > Labels: TriagePending > Attachments: OzoneFsTrash.patch > > > The aim of this jira to add the trash feature capability in ozone that will > help prevent accidental deletion of files and directories. When a file gets > deleted in Ozone, the file is not immediately expelled from Ozone but will be > moved to trash. Similarly as in HDFS, to immediately delete a file, a > skipTrash option will be provided. The jira also aims to integrate the Ozone > FS Trash to the Trash API developed as a part of HDDS-2416. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3350) Ozone Retry Policy Improvements
[ https://issues.apache.org/jira/browse/HDDS-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3350: Labels: Triaged (was: ) > Ozone Retry Policy Improvements > --- > > Key: HDDS-3350 > URL: https://issues.apache.org/jira/browse/HDDS-3350 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Labels: Triaged > Attachments: Retry Behaviour in Ozone Client.pdf, Retry Behaviour in > Ozone Client_Updated.pdf, Retry Behaviour in Ozone Client_Updated_2.pdf > > > Currently any ozone client request can spend a huge amount of time in retries > and ozone client can retry its requests very aggressively. The waiting time > can thus be very high before a client request fails. Further aggressive > retries by ratis client used by ozone can bog down a ratis pipeline leader. > The Jira aims to make changes to the current retry behavior in Ozone client. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3342) Ozone admin shell commands do not print or log exceptions on failures
[ https://issues.apache.org/jira/browse/HDDS-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3342: Labels: TriagePending (was: ) > Ozone admin shell commands do not print or log exceptions on failures > - > > Key: HDDS-3342 > URL: https://issues.apache.org/jira/browse/HDDS-3342 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone CLI >Affects Versions: 0.5.0 >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: TriagePending > > Ozone shell commands such as `ozone admin container create` does not print > any information on failure which makes it very difficult to debug the > underlying issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3329) Ozone cluster expansion: Block deletion mismatch
[ https://issues.apache.org/jira/browse/HDDS-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3329: Target Version/s: 0.6.0 Labels: TriagePending (was: ) > Ozone cluster expansion: Block deletion mismatch > > > Key: HDDS-3329 > URL: https://issues.apache.org/jira/browse/HDDS-3329 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.4.1 >Reporter: Li Cheng >Assignee: Lokesh Jain >Priority: Major > Labels: TriagePending > Attachments: hadoop-ozoneadmin-scm.log, ozone-scm-dn-log.tar.gz > > > SCM logs keep printing this when we expand Ozone cluster with more datanodes. > > 2020-04-02 19:45:42,745 > [EventQueue-PendingDeleteStatusForPendingDeleteHandler] INFO > org.apache.hadoop.hdds.scm.block.SCMBlockDeletingService: Block deletion > txnID mismatch in datanode 1eacbd89-a835-438e-aa4b-5bc78adb7c8c for > containerID 314. Datanode delete txnID: 0, SCM txnID: 1208 > 2020-04-02 19:45:42,745 > [EventQueue-PendingDeleteStatusForPendingDeleteHandler] INFO > org.apache.hadoop.hdds.scm.block.SCMBlockDeletingService: Block deletion > txnID mismatch in datanode 1eacbd89-a835-438e-aa4b-5bc78adb7c8c for > containerID 351. Datanode delete txnID: 0, SCM txnID: 662 > 2020-04-02 19:45:42,745 > [EventQueue-PendingDeleteStatusForPendingDeleteHandler] INFO > org.apache.hadoop.hdds.scm.block.SCMBlockDeletingService: Block deletion > txnID mismatch in datanode 1eacbd89-a835-438e-aa4b-5bc78adb7c8c for > containerID 352. Datanode delete txnID: 0, SCM txnID: 1085 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3325) Handle Resource Unavailable exception in OM HA
[
https://issues.apache.org/jira/browse/HDDS-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3325:
Target Version/s: 0.6.0
Labels: TriagePending (was: )
> Handle Resource Unavailable exception in OM HA
> --
>
> Key: HDDS-3325
> URL: https://issues.apache.org/jira/browse/HDDS-3325
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: HA, Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: TriagePending
>
> Right now, when the future fails with an exception, we send that exception to
> the client, and retry with a new server. but when using ratis server when
> resource unavailable exception future fails with exceptionally. So, in this
> case we need to wrap the exception and retry to the same server with some
> retry policy like MultiLinearRandomRetry or some retry policy.
> {code:java}
> try {
> raftClientReply = server.submitClientRequestAsync(raftClientRequest)
> .get();
> } catch (Exception ex) {
> throw new ServiceException(ex.getMessage(), ex);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3396) Add metrics for stateMachine cache count and size in bytes
[ https://issues.apache.org/jira/browse/HDDS-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3396: Fix Version/s: (was: 0.6.0) Target Version/s: 0.7.0 > Add metrics for stateMachine cache count and size in bytes > -- > > Key: HDDS-3396 > URL: https://issues.apache.org/jira/browse/HDDS-3396 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Shashikant Banerjee >Assignee: runzhiwang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3286) BasicOzoneFileSystem support batchDelete
[ https://issues.apache.org/jira/browse/HDDS-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3286: Target Version/s: 0.6.0 Labels: Triaged pull-request-available (was: pull-request-available) > BasicOzoneFileSystem support batchDelete > - > > Key: HDDS-3286 > URL: https://issues.apache.org/jira/browse/HDDS-3286 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Filesystem >Reporter: mingchao zhao >Assignee: mingchao zhao >Priority: Major > Labels: Triaged, pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently delete file is to get all the keys in the directory, and then > delete one by one. And the same with rename. This makes for poor performance. > By tested the deletion path with 100,000 files, which took 3718.70 sec. > And rename it took 7327.936. > HDDS-2939 can able to optimize this part, but at present the hdds-2939 is > slow and still a long way to go. So we optimized the batch operation based on > the current interface. We were able to get better performance with this PR > before the hdds-2939 came in. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3277) Datanodes do not close pipeline when pipeline directory is deleted.
[
https://issues.apache.org/jira/browse/HDDS-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3277:
Target Version/s: 0.6.0
Labels: MiniOzoneChaosCluster Triaged (was:
MiniOzoneChaosCluster)
> Datanodes do not close pipeline when pipeline directory is deleted.
> ---
>
> Key: HDDS-3277
> URL: https://issues.apache.org/jira/browse/HDDS-3277
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.6.0
>Reporter: Mukul Kumar Singh
>Priority: Major
> Labels: MiniOzoneChaosCluster, Triaged
>
> First the pipeline was deleted
> {code}
> 2020-03-25 19:44:22,669 [pool-22-thread-1] INFO failure.Failures
> (FailureManager.java:fail(49)) - failing with, DeletePipelineFailure
> 2020-03-25 19:44:22,669 [pool-22-thread-1] INFO failure.Failures
> (Failures.java:fail(118)) - deleteing pipeline directory
> /tmp/chaos-2020-03-25-19-42-52-IST/MiniOzoneClusterImpl-ef9b224f-a403-4e9b-a27a-ed38f46700
> c5/datanode-0/data/ratis/c4275846-2a44-4f53-b00d-c95a81785df9
> 2020-03-25 19:44:22,679 [pool-22-thread-1] INFO failure.Failures
> (Failures.java:fail(118)) - deleteing pipeline directory
> /tmp/chaos-2020-03-25-19-42-52-IST/MiniOzoneClusterImpl-ef9b224f-a403-4e9b-a27a-ed38f46700
> c5/datanode-3/data/ratis/c4275846-2a44-4f53-b00d-c95a81785df9
> 2020-03-25 19:44:22,681 [pool-22-thread-1] INFO failure.Failures
> (Failures.java:fail(118)) - deleteing pipeline directory
> /tmp/chaos-2020-03-25-19-42-52-IST/MiniOzoneClusterImpl-ef9b224f-a403-4e9b-a27a-ed38f46700
> c5/datanode-5/data/ratis/c4275846-2a44-4f53-b00d-c95a81785df9
> {code}
> However no pipeline failure handling was issued to SCM.
> {code}
> 2020-03-25 19:44:24,532
> [b5d165bc-d2b3-497c-ae38-10f649674a3f@group-C95A81785DF9-StateMachineUpdater]
> ERROR ratis.ContainerStateMachine
> (ContainerStateMachine.java:takeSnapshot(302)) - group-C95A81785DF9: Failed
> to write snapshot at:(t:1, i:2037) file
> /tmp/chaos-2020-03-25-19-42-52-IST/MiniOzoneClusterImpl-ef9b224f-a403-4e9b-a27a-ed38f46700c5/datanode-3/data/ratis/c4275846-2a44-4f53-b00d-c95a81785df9/sm/snapshot.1_2037
> 2020-03-25 19:44:24,532
> [b5d165bc-d2b3-497c-ae38-10f649674a3f@group-C95A81785DF9-StateMachineUpdater]
> ERROR impl.StateMachineUpdater (StateMachineUpdater.java:takeSnapshot(269)) -
> b5d165bc-d2b3-497c-ae38-10f649674a3f@group-C95A81785DF9-StateMachineUpdater:
> Failed to take snapshot
> java.io.FileNotFoundException:
> /tmp/chaos-2020-03-25-19-42-52-IST/MiniOzoneClusterImpl-ef9b224f-a403-4e9b-a27a-ed38f46700c5/datanode-3/data/ratis/c4275846-2a44-4f53-b00d-c95a81785df9/sm/snapshot.1_2037
> (No such file or directory)
> at java.io.FileOutputStream.open0(Native Method)
> at java.io.FileOutputStream.open(FileOutputStream.java:270)
> at java.io.FileOutputStream.(FileOutputStream.java:213)
> at java.io.FileOutputStream.(FileOutputStream.java:162)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.takeSnapshot(ContainerStateMachine.java:296)
> at
> org.apache.ratis.server.impl.StateMachineUpdater.takeSnapshot(StateMachineUpdater.java:258)
> at
> org.apache.ratis.server.impl.StateMachineUpdater.checkAndTakeSnapshot(StateMachineUpdater.java:250)
> at
> org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:169)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3274) Improve documentation for specifying serviceId
[ https://issues.apache.org/jira/browse/HDDS-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3274: Labels: Triaged (was: ) > Improve documentation for specifying serviceId > -- > > Key: HDDS-3274 > URL: https://issues.apache.org/jira/browse/HDDS-3274 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: documentation >Affects Versions: 0.5.0 >Reporter: Dinesh Chitlangia >Priority: Major > Labels: Triaged > > Discovered by [~xyao] when testing 0.5.0-beta rc2 > > The document on specifying service/ID can be improved. More > specifically, the URI should give examples for the Service ID in HA. > Currently, it only mentions host/port. > ozone sh vol create /vol1 > Service ID or host name must not be omitted when ozone.om.service.ids is > defined. > bash-4.2$ ozone sh vol create --help > Usage: ozone sh volume create [-hV] [--root] [-q=] [-u=] > > Creates a volume for the specified user > URI of the volume. > Ozone URI could start with o3:// or without > prefix. URI > may contain the host and port of the OM server. > Both are > optional. If they are not specified it will be > identified from the config files. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3275) scmcli reports incorrect details
[
https://issues.apache.org/jira/browse/HDDS-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3275:
Target Version/s: 0.7.0 (was: 0.6.0)
> scmcli reports incorrect details
>
>
> Key: HDDS-3275
> URL: https://issues.apache.org/jira/browse/HDDS-3275
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Tools
>Affects Versions: 0.5.0
>Reporter: Dinesh Chitlangia
>Priority: Major
>
> Reported by [~xyao]
>
> ozone scmcli container list seems report incorrect numberOfKeys and
> usedBytes
> Also, container owner is set as the current leader om(om3), should we use
> the om service id here instead?
> bash-4.2$ ozone scmcli container list
> {
> "state" : "OPEN",
> "replicationFactor" : "THREE",
> "replicationType" : "RATIS",
> "usedBytes" : 3813,
> "numberOfKeys" : 1,
> ...
> bash-4.2$ ozone sh key list o3://id1/vol1/bucket1/
> {
> "volumeName" : "vol1",
> "bucketName" : "bucket1",
> "name" : "k1",
> "dataSize" : 3813,
> "creationTime" : "2020-03-23T03:23:30.670Z",
> "modificationTime" : "2020-03-23T03:23:33.207Z",
> "replicationType" : "RATIS",
> "replicationFactor" : 3
> }
> {
> "volumeName" : "vol1",
> "bucketName" : "bucket1",
> "name" : "k2",
> "dataSize" : 3813,
> "creationTime" : "2020-03-23T03:18:46.735Z",
> "modificationTime" : "2020-03-23T03:20:15.005Z",
> "replicationType" : "RATIS",
> "replicationFactor" : 3
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
