[jira] [Commented] (HDDS-2283) Container Creation on datanodes take around 300ms due to rocksdb creation
[ https://issues.apache.org/jira/browse/HDDS-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953232#comment-16953232 ] Siddharth Wagle commented on HDDS-2283: --- [~aengineer] Yes the follow-up Jira will not be blindly taken up without figuring out if 10s of RocksDBs sharing a disk vs 1 RocksDB with 10 tables in a single RocksDB which one is better/worse. I took this up as a low hanging fruit, agree with not focusing on micro-benchmarks comment. This was just a curiosity / exploratory effort from me that took all of 20 mins including the fix. > Container Creation on datanodes take around 300ms due to rocksdb creation > - > > Key: HDDS-2283 > URL: https://issues.apache.org/jira/browse/HDDS-2283 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Major > Labels: pull-request-available > Attachments: HDDS-2283.00.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Container Creation on datanodes take around 300ms due to rocksdb creation. > Rocksdb creation is taking a considerable time and this needs to be optimized. > Creating a rocksdb per disk should be enough and each container can be table > inside the rocksdb. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2283) Container Creation on datanodes take around 300ms due to rocksdb creation
[ https://issues.apache.org/jira/browse/HDDS-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953181#comment-16953181 ] Anu Engineer commented on HDDS-2283: I am fine this this change.. but the follow up JIRA makes no sense to me, my earlier deleted comment was about that. > Container Creation on datanodes take around 300ms due to rocksdb creation > - > > Key: HDDS-2283 > URL: https://issues.apache.org/jira/browse/HDDS-2283 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Major > Labels: pull-request-available > Attachments: HDDS-2283.00.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Container Creation on datanodes take around 300ms due to rocksdb creation. > Rocksdb creation is taking a considerable time and this needs to be optimized. > Creating a rocksdb per disk should be enough and each container can be table > inside the rocksdb. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2283) Container Creation on datanodes take around 300ms due to rocksdb creation
[ https://issues.apache.org/jira/browse/HDDS-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953179#comment-16953179 ] Anu Engineer commented on HDDS-2283: I really doubt if this is a bottleneck. I am not disagreeing with the measurement. I am just saying in the big picture of writing 5GB of data, optimizing for this 300 ms buys you nothing. if you are optimizing for micro-benchmarks, what we will end of building is a system that is optimized for these benchmarks; not something that is really needed. > Container Creation on datanodes take around 300ms due to rocksdb creation > - > > Key: HDDS-2283 > URL: https://issues.apache.org/jira/browse/HDDS-2283 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Major > Labels: pull-request-available > Attachments: HDDS-2283.00.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Container Creation on datanodes take around 300ms due to rocksdb creation. > Rocksdb creation is taking a considerable time and this needs to be optimized. > Creating a rocksdb per disk should be enough and each container can be table > inside the rocksdb. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2283) Container Creation on datanodes take around 300ms due to rocksdb creation
[ https://issues.apache.org/jira/browse/HDDS-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952327#comment-16952327 ] Siddharth Wagle commented on HDDS-2283: --- I will create a PR with code cleanup and UT completed. > Container Creation on datanodes take around 300ms due to rocksdb creation > - > > Key: HDDS-2283 > URL: https://issues.apache.org/jira/browse/HDDS-2283 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Major > Attachments: HDDS-2283.00.patch > > > Container Creation on datanodes take around 300ms due to rocksdb creation. > Rocksdb creation is taking a considerable time and this needs to be optimized. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2283) Container Creation on datanodes take around 300ms due to rocksdb creation
[
https://issues.apache.org/jira/browse/HDDS-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952325#comment-16952325
]
Siddharth Wagle commented on HDDS-2283:
---
[~msingh] Attaching a speculative patch can you let me know your thoughts?
With the patch:
{code}
2019-10-15 14:35:40,290 INFO volume.HddsVolume (HddsVolume.java:(176)) -
Creating Volume:
/var/folders/7y/d3vtnjg502sgppd08pj0j0j4gp/T/junit6254259191707885578/hdds
of storage type : null and capacity : 500068036608
2019-10-15 14:35:40,949 INFO utils.MetadataStoreBuilder
(MetadataStoreBuilder.java:build(149)) - Time before create: 84
2019-10-15 14:35:40,950 INFO utils.RocksDBStore (RocksDBStore.java:(68))
- Time to load library: 0
2019-10-15 14:35:40,958 INFO utils.RocksDBStore (RocksDBStore.java:(75))
- Time to open: 7
2019-10-15 14:35:40,958 INFO helpers.KeyValueContainerUtil
(KeyValueContainerUtil.java:createContainerMetaData(85)) - Total time to
create: 98
2019-10-15 14:35:41,013 WARN util.NativeCodeLoader
(NativeCodeLoader.java:(60)) - Unable to load native-hadoop library for
your platform... using builtin-java classes where applicable
2019-10-15 14:35:41,015 INFO utils.MetadataStoreBuilder
(MetadataStoreBuilder.java:build(149)) - Time before create: 0
2019-10-15 14:35:41,015 INFO utils.RocksDBStore (RocksDBStore.java:(68))
- Time to load library: 0
2019-10-15 14:35:41,021 INFO utils.RocksDBStore (RocksDBStore.java:(75))
- Time to open: 6
2019-10-15 14:35:41,021 INFO helpers.KeyValueContainerUtil
(KeyValueContainerUtil.java:createContainerMetaData(85)) - Total time to
create: 7
{code}
> Container Creation on datanodes take around 300ms due to rocksdb creation
> -
>
> Key: HDDS-2283
> URL: https://issues.apache.org/jira/browse/HDDS-2283
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Mukul Kumar Singh
>Assignee: Siddharth Wagle
>Priority: Major
>
> Container Creation on datanodes take around 300ms due to rocksdb creation.
> Rocksdb creation is taking a considerable time and this needs to be optimized.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2283) Container Creation on datanodes take around 300ms due to rocksdb creation
[
https://issues.apache.org/jira/browse/HDDS-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952262#comment-16952262
]
Siddharth Wagle commented on HDDS-2283:
---
[~msingh] In my unit test, the obvious bottleneck is time to create the RocksDB
options object which is doing a native call:
{code}
2019-10-15 13:20:10,714 INFO utils.MetadataStoreBuilder
(MetadataStoreBuilder.java:build(124)) - Time before create, load options: 81
2019-10-15 13:20:10,715 INFO utils.RocksDBStore (RocksDBStore.java:(68))
- Time to load library: 0
2019-10-15 13:20:10,723 INFO utils.RocksDBStore (RocksDBStore.java:(75))
- Time to open: 8
2019-10-15 13:20:10,723 INFO helpers.KeyValueContainerUtil
(KeyValueContainerUtil.java:createContainerMetaData(85)) - Total time to
create: {}95
{code}
> Container Creation on datanodes take around 300ms due to rocksdb creation
> -
>
> Key: HDDS-2283
> URL: https://issues.apache.org/jira/browse/HDDS-2283
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Mukul Kumar Singh
>Assignee: Siddharth Wagle
>Priority: Major
>
> Container Creation on datanodes take around 300ms due to rocksdb creation.
> Rocksdb creation is taking a considerable time and this needs to be optimized.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
