[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244739=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244739 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 20/May/19 02:23 Start Date: 20/May/19 02:23 Worklog Time Spent: 10m Work Description: arp7 commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244739) Time Spent: 12h (was: 11h 50m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 12h > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244737=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244737 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 20/May/19 02:22 Start Date: 20/May/19 02:22 Worklog Time Spent: 10m Work Description: arp7 commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r285413624 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStore.java ## @@ -44,6 +44,7 @@ */ Table getTable(String name) throws IOException; + Review comment: Committing this for now since Anu +1'ed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244737) Time Spent: 11h 40m (was: 11.5h) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 11h 40m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244738=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244738 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 20/May/19 02:22 Start Date: 20/May/19 02:22 Worklog Time Spent: 10m Work Description: arp7 commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-493818528 +1 LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244738) Time Spent: 11h 50m (was: 11h 40m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 11h 50m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244736=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244736 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 20/May/19 02:21 Start Date: 20/May/19 02:21 Worklog Time Spent: 10m Work Description: arp7 commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r285413487 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java ## @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.hadoop.utils.db.cache; + +import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.classification.InterfaceStability.Evolving; + +/** + * Cache used for RocksDB tables. + * @param + * @param + */ + +@Private +@Evolving +public interface TableCache OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 11.5h > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244735=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244735
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 20/May/19 02:20
Start Date: 20/May/19 02:20
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285413415
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -106,6 +142,40 @@ public void close() throws Exception {
}
+ @Override
+ public void addCacheEntry(CacheKey cacheKey,
+ CacheValue cacheValue) {
+// This will override the entry if there is already entry for this key.
+cache.put(cacheKey, cacheValue);
+ }
+
+
+ @Override
+ public void cleanupCache(long epoch) {
Review comment:
We can revisit this later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244735)
Time Spent: 11h 20m (was: 11h 10m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 11h 20m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244519=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244519 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 18/May/19 10:05 Start Date: 18/May/19 10:05 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r285336556 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStore.java ## @@ -44,6 +44,7 @@ */ Table getTable(String name) throws IOException; + Review comment: Nit: Space only change? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244519) Time Spent: 11h 10m (was: 11h) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 11h 10m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244518=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244518 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 18/May/19 10:05 Start Date: 18/May/19 10:05 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r285336643 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/Table.java ## @@ -97,6 +102,28 @@ void putWithBatch(BatchOperation batch, KEY key, VALUE value) */ String getName() throws IOException; + /** + * Add entry to the table cache. + * + * If the cacheKey already exists, it will override the entry. + * @param cacheKey + * @param cacheValue + */ Review comment: thx makes sense. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244518) Time Spent: 11h (was: 10h 50m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 11h > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244401=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244401
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:40
Start Date: 17/May/19 22:40
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285305076
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -106,6 +142,40 @@ public void close() throws Exception {
}
+ @Override
+ public void addCacheEntry(CacheKey cacheKey,
+ CacheValue cacheValue) {
+// This will override the entry if there is already entry for this key.
+cache.put(cacheKey, cacheValue);
+ }
+
+
+ @Override
+ public void cleanupCache(long epoch) {
Review comment:
This is added because, in actual usage of classes, we get Table reference,
and they can simply call table.cleanupCache(). Added getCache() is used only
for testing, so declared it as not a public method and annotated with
VisibleForTesting. And usage of this will be in the code, once double-buffer
integration is completed.
Let me know if you still want to be removed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244401)
Time Spent: 10h 50m (was: 10h 40m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h 50m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244392=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244392 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 17/May/19 22:28 Start Date: 17/May/19 22:28 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-493617081 Thank You @arp7 for the review. I have addressed the review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244392) Time Spent: 10h 40m (was: 10.5h) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10h 40m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244391=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244391
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:28
Start Date: 17/May/19 22:28
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285306662
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/PartialTableCache.java
##
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import java.util.Iterator;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+/**
+ * Cache implementation for the table, this cache is partial cache, this will
+ * be cleaned up, after entries are flushed to DB.
+ */
+@Private
+@Evolving
+public class PartialTableCache implements TableCache
{
+
+ private final ConcurrentHashMap cache;
+ private final TreeSet> epochEntries;
+ private ExecutorService executorService;
+
+
+
+ public PartialTableCache() {
+cache = new ConcurrentHashMap<>();
+epochEntries = new TreeSet<>();
+// Created a singleThreadExecutor, so one cleanup will be running at a
+// time.
+executorService = Executors.newSingleThreadExecutor();
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244391)
Time Spent: 10.5h (was: 10h 20m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244383=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244383 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 17/May/19 22:20 Start Date: 17/May/19 22:20 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r285305397 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java ## @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.hadoop.utils.db.cache; + +import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.classification.InterfaceStability.Evolving; + +/** + * Cache used for RocksDB tables. + * @param + * @param + */ + +@Private +@Evolving +public interface TableCache * @param */ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244383) Time Spent: 10h 20m (was: 10h 10m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10h 20m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244381=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244381
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:18
Start Date: 17/May/19 22:18
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285305076
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -106,6 +142,40 @@ public void close() throws Exception {
}
+ @Override
+ public void addCacheEntry(CacheKey cacheKey,
+ CacheValue cacheValue) {
+// This will override the entry if there is already entry for this key.
+cache.put(cacheKey, cacheValue);
+ }
+
+
+ @Override
+ public void cleanupCache(long epoch) {
Review comment:
This is added because, in actual usage of classes, we get Table reference,
and they can simply call table.cleanupCache(). Added getCache() only for
testing, so declared it as not a public method and annotated with
VisibleForTesting. This will be used after double-buffer integration completed.
Let me know if you still want to be removed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244381)
Time Spent: 10h 10m (was: 10h)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h 10m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244378=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244378
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:15
Start Date: 17/May/19 22:15
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285304435
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -69,8 +79,34 @@ public boolean isEmpty() throws IOException {
return rawTable.isEmpty();
}
+ /**
+ * Returns the value mapped to the given key in byte array or returns null
+ * if the key is not found.
+ *
+ * Caller's of this method should use synchronization mechanism, when
+ * accessing. First it will check from cache, if it has entry return the
+ * value, otherwise get from the RocksDB table.
+ *
+ * @param key metadata key
+ * @return VALUE
+ * @throws IOException
+ */
@Override
public VALUE get(KEY key) throws IOException {
+// Here the metadata lock will guarantee that cache is not updated for same
+// key during get key.
+CacheValue< VALUE > cacheValue = cache.get(new CacheKey<>(key));
+if (cacheValue == null) {
+ // If no cache for the table or if it does not exist in cache get from
+ // RocksDB table.
+ return getFromTable(key);
+} else {
+ // We have a value in cache, return the value.
+ return cacheValue.getValue();
Review comment:
Yes actual value in CacheValue can be null.
**Implementation of CacheValue getValue**
public VALUE getValue() {
return value.orNull();
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244378)
Time Spent: 10h (was: 9h 50m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244362=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244362
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:14
Start Date: 17/May/19 22:14
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285304280
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -31,13 +37,16 @@
*/
public class TypedTable implements Table {
- private Table rawTable;
+ private final Table rawTable;
+
+ private final CodecRegistry codecRegistry;
- private CodecRegistry codecRegistry;
+ private final Class keyType;
Review comment:
Used by TableIterator, to deserialize actual table byte[] data to particular
key value format it iwill be used.
Note: This is not added by this change, only converted them to final.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244362)
Time Spent: 9h 50m (was: 9h 40m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 50m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244355=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244355
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:05
Start Date: 17/May/19 22:05
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285301711
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -69,8 +79,34 @@ public boolean isEmpty() throws IOException {
return rawTable.isEmpty();
}
+ /**
+ * Returns the value mapped to the given key in byte array or returns null
+ * if the key is not found.
+ *
+ * Caller's of this method should use synchronization mechanism, when
+ * accessing. First it will check from cache, if it has entry return the
+ * value, otherwise get from the RocksDB table.
+ *
+ * @param key metadata key
+ * @return VALUE
+ * @throws IOException
+ */
@Override
public VALUE get(KEY key) throws IOException {
+// Here the metadata lock will guarantee that cache is not updated for same
+// key during get key.
+CacheValue< VALUE > cacheValue = cache.get(new CacheKey<>(key));
+if (cacheValue == null) {
+ // If no cache for the table or if it does not exist in cache get from
+ // RocksDB table.
+ return getFromTable(key);
+} else {
+ // We have a value in cache, return the value.
+ return cacheValue.getValue();
Review comment:
cacheValue.getValue() can be a null to indicate deletion, correct?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244355)
Time Spent: 9h 20m (was: 9h 10m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244357=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244357
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:05
Start Date: 17/May/19 22:05
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285302353
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/PartialTableCache.java
##
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import java.util.Iterator;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+/**
+ * Cache implementation for the table, this cache is partial cache, this will
+ * be cleaned up, after entries are flushed to DB.
+ */
+@Private
+@Evolving
+public class PartialTableCache implements TableCache
{
+
+ private final ConcurrentHashMap cache;
+ private final TreeSet> epochEntries;
+ private ExecutorService executorService;
+
+
+
+ public PartialTableCache() {
+cache = new ConcurrentHashMap<>();
+epochEntries = new TreeSet<>();
+// Created a singleThreadExecutor, so one cleanup will be running at a
+// time.
+executorService = Executors.newSingleThreadExecutor();
Review comment:
Can you use ThreadFactory here to give the executor thread a descriptive
name?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244357)
Time Spent: 9h 40m (was: 9.5h)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244356=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244356
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:05
Start Date: 17/May/19 22:05
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285301960
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -106,6 +142,40 @@ public void close() throws Exception {
}
+ @Override
+ public void addCacheEntry(CacheKey cacheKey,
+ CacheValue cacheValue) {
+// This will override the entry if there is already entry for this key.
+cache.put(cacheKey, cacheValue);
+ }
+
+
+ @Override
+ public void cleanupCache(long epoch) {
Review comment:
We can remove this function I think. Caller can do
`table.getCache().cleanup()`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244356)
Time Spent: 9.5h (was: 9h 20m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244358=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244358 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 17/May/19 22:05 Start Date: 17/May/19 22:05 Worklog Time Spent: 10m Work Description: arp7 commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r285302472 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java ## @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.hadoop.utils.db.cache; + +import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.classification.InterfaceStability.Evolving; + +/** + * Cache used for RocksDB tables. + * @param + * @param + */ + +@Private +@Evolving +public interface TableCache OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 9h 40m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244354=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244354
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 17/May/19 22:05
Start Date: 17/May/19 22:05
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r285301379
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -31,13 +37,16 @@
*/
public class TypedTable implements Table {
- private Table rawTable;
+ private final Table rawTable;
+
+ private final CodecRegistry codecRegistry;
- private CodecRegistry codecRegistry;
+ private final Class keyType;
Review comment:
@bharatviswa504 why do we need the keyType and valueType fields?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244354)
Time Spent: 9h 20m (was: 9h 10m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=244215=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244215 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 17/May/19 19:00 Start Date: 17/May/19 19:00 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-493563512 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 58 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 84 | Maven dependency ordering for branch | | +1 | mvninstall | 576 | trunk passed | | +1 | compile | 238 | trunk passed | | +1 | checkstyle | 61 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 1015 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 149 | trunk passed | | 0 | spotbugs | 288 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 534 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for patch | | +1 | mvninstall | 533 | the patch passed | | +1 | compile | 256 | the patch passed | | +1 | javac | 256 | the patch passed | | +1 | checkstyle | 68 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 774 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 153 | the patch passed | | +1 | findbugs | 584 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 188 | hadoop-hdds in the patch failed. | | -1 | unit | 1100 | hadoop-ozone in the patch failed. | | +1 | asflicense | 36 | The patch does not generate ASF License warnings. | | | | 6581 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestBlockOutputStream | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | | hadoop.ozone.client.rpc.TestKeyInputStream | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-798/9/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/798 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux deac2877b50b 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 4cb3da6 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/9/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/9/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/9/testReport/ | | Max. process+thread count | 3554 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/server-scm hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/9/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244215) Time Spent: 9h 10m (was: 9h) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 >
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=243107=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243107 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 16/May/19 04:21 Start Date: 16/May/19 04:21 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-492910899 > Seems like cache.put is never called except testcases, cache will always be empty. This jira add's Table Cache, this is not integrated to OM Code, in further jira's it will be integrated, and we shall call this API put. Jira description also has the same information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243107) Time Spent: 9h (was: 8h 50m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 9h > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=243100=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243100 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 16/May/19 03:51 Start Date: 16/May/19 03:51 Worklog Time Spent: 10m Work Description: chinphing commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-492905610 Seems like cache.put is never called except testcases, cache will always be empty. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243100) Time Spent: 8h 50m (was: 8h 40m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 8h 50m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=243095=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243095
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 16/May/19 03:47
Start Date: 16/May/19 03:47
Worklog Time Spent: 10m
Work Description: chinphing commented on issue #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#issuecomment-492905610
Seems like addCacheEntry is never called, cache will always be empty.
`
@Override
public void addCacheEntry(CacheKey cacheKey,
CacheValue cacheValue) {
// This will override the entry if there is already entry for this key.
cache.put(cacheKey, cacheValue);
}
`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243095)
Time Spent: 8.5h (was: 8h 20m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8.5h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=243097=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243097
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 16/May/19 03:47
Start Date: 16/May/19 03:47
Worklog Time Spent: 10m
Work Description: chinphing commented on issue #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#issuecomment-492905610
Seems like addCacheEntry is never called, cache will always be empty.
`@Override
public void addCacheEntry(CacheKey cacheKey,
CacheValue cacheValue) {
// This will override the entry if there is already entry for this key.
cache.put(cacheKey, cacheValue);
}`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243097)
Time Spent: 8h 40m (was: 8.5h)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242923=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242923 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 15/May/19 22:53 Start Date: 15/May/19 22:53 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-492853658 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 35 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 62 | Maven dependency ordering for branch | | +1 | mvninstall | 387 | trunk passed | | +1 | compile | 204 | trunk passed | | +1 | checkstyle | 52 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 811 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 125 | trunk passed | | 0 | spotbugs | 239 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 416 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for patch | | +1 | mvninstall | 392 | the patch passed | | +1 | compile | 193 | the patch passed | | +1 | javac | 193 | the patch passed | | +1 | checkstyle | 55 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 664 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 126 | the patch passed | | +1 | findbugs | 431 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 138 | hadoop-hdds in the patch failed. | | -1 | unit | 1068 | hadoop-ozone in the patch failed. | | +1 | asflicense | 42 | The patch does not generate ASF License warnings. | | | | 5323 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdds.scm.pipeline.TestPipelineClose | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-798/8/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/798 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 97f2fd6b2c5c 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 77170e7 | | Default Java | 1.8.0_191 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/8/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/8/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/8/testReport/ | | Max. process+thread count | 4479 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/server-scm hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/8/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 242923) Time Spent: 8h 20m (was: 8h 10m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 8h 20m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242880=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242880
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 15/May/19 21:38
Start Date: 15/May/19 21:38
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284462072
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -71,6 +96,27 @@ public boolean isEmpty() throws IOException {
@Override
public VALUE get(KEY key) throws IOException {
+// Here the metadata lock will guarantee that cache is not updated for same
+// key during get key.
+if (cache != null) {
+ CacheValue cacheValue = cache.get(new CacheKey<>(key));
+ if (cacheValue == null) {
+return getFromTable(key);
+ } else {
+// Doing this because, if the Cache Value Last operation is deleted
+// means it will eventually removed from DB. So, we should return null.
+if (cacheValue.getLastOperation() != CacheValue.OperationType.DELETED)
{
+ return cacheValue.getValue();
+} else {
+ return null;
+}
+ }
+} else {
+ return getFromTable(key);
Review comment:
Understood the comment, updated the code to remove getTable in multiple
places.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242880)
Time Spent: 8h 10m (was: 8h)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h 10m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242878=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242878
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 15/May/19 21:37
Start Date: 15/May/19 21:37
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284461812
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/PartialTableCache.java
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import java.util.Iterator;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+
+
+/**
+ * This is used for the tables where we don't want to cache entire table in
+ * in-memory.
+ */
+@Private
+@Evolving
+public class PartialTableCache
+implements TableCache{
+
+ private final ConcurrentHashMap cache;
+ private final TreeSet> epochEntries;
+ private ExecutorService executorService;
+
+
+
+ public PartialTableCache() {
+cache = new ConcurrentHashMap<>();
+epochEntries = new TreeSet>();
+// Created a singleThreadExecutor, so one cleanup will be running at a
+// time.
+executorService = Executors.newSingleThreadExecutor();
+ }
+
+ @Override
+ public CACHEVALUE get(CACHEKEY cachekey) {
+return cache.get(cachekey);
+ }
+
+ @Override
+ public void put(CACHEKEY cacheKey, CACHEVALUE value) {
+cache.put(cacheKey, value);
+CacheValue cacheValue = (CacheValue) cache.get(cacheKey);
+epochEntries.add(new EpochEntry<>(cacheValue.getEpoch(), cacheKey));
+ }
+
+ @Override
+ public void cleanup(long epoch) {
+executorService.submit(() -> evictCache(epoch));
+ }
+
+ @Override
+ public int size() {
+return cache.size();
+ }
+
+ private void evictCache(long epoch) {
Review comment:
Yes Key will be evicted once double buffer flushes to disk.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242878)
Time Spent: 8h (was: 7h 50m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242874=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242874 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 15/May/19 21:31 Start Date: 15/May/19 21:31 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-492831553 Thank You @arp7 for offline discussion. I have addressed review the comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 242874) Time Spent: 7h 50m (was: 7h 40m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 7h 50m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242871=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242871
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 15/May/19 21:27
Start Date: 15/May/19 21:27
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284458211
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
##
@@ -245,42 +247,50 @@ protected DBStoreBuilder
addOMTablesAndCodecs(DBStoreBuilder builder) {
*/
protected void initializeOmTables() throws IOException {
userTable =
-this.store.getTable(USER_TABLE, String.class, VolumeList.class);
+this.store.getTable(USER_TABLE, String.class, VolumeList.class,
Review comment:
Done. Now caller's no need to specify CacheType.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242871)
Time Spent: 7h 40m (was: 7.5h)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242870=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242870
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 15/May/19 21:26
Start Date: 15/May/19 21:26
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284458025
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+/**
+ * Cache used for RocksDB tables.
+ * @param
+ * @param
+ */
+
+@Private
+@Evolving
+public interface TableCache {
+
+ /**
+ * Return the value for the key if it is present, otherwise return null.
+ * @param cacheKey
+ * @return CACHEVALUE
+ */
+ CACHEVALUE get(CACHEKEY cacheKey);
+
+ /**
+ * Add an entry to the cache, if the key already exists it overrides.
+ * @param cacheKey
+ * @param value
+ */
+ void put(CACHEKEY cacheKey, CACHEVALUE value);
+
+ /**
+ * Removes all the entries from the cache which are having epoch value less
+ * than or equal to specified epoch value.
+ * @param epoch
+ */
+ void cleanup(long epoch);
+
+ /**
+ * Return the size of the cache.
+ * @return size
+ */
+ int size();
+
+ /**
+ * Defines type of cache need to be used by OM RocksDB tables.
+ */
+ enum CACHETYPE {
Review comment:
Removed FullCache, and have the only implementation for TableCache i.e
PartialCache.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242870)
Time Spent: 7.5h (was: 7h 20m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242869=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242869 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 15/May/19 21:25 Start Date: 15/May/19 21:25 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r284457687 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/FullTableCache.java ## @@ -0,0 +1,67 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.utils.db.cache; + +import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.classification.InterfaceStability.Evolving; + +import java.util.concurrent.ConcurrentHashMap; + +/** + * This is the full table cache, where it uses concurrentHashMap internally, + * and does not do any evict or cleanup. This full table cache need to be + * used by tables where we want to cache the entire table with out any + * cleanup to the cache + * @param + * @param + */ + +@Private +@Evolving +public class FullTableCache OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 7h 20m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242868=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242868
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 15/May/19 21:25
Start Date: 15/May/19 21:25
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284457474
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -31,22 +38,40 @@
*/
public class TypedTable implements Table {
- private Table rawTable;
+ private final Table rawTable;
+
+ private final CodecRegistry codecRegistry;
- private CodecRegistry codecRegistry;
+ private final Class keyType;
- private Class keyType;
+ private final Class valueType;
- private Class valueType;
+ private final TableCache, CacheValue> cache;
public TypedTable(
Table rawTable,
CodecRegistry codecRegistry, Class keyType,
Class valueType) {
+this(rawTable, codecRegistry, keyType, valueType,
+null);
+ }
+
+
+ public TypedTable(
+ Table rawTable,
+ CodecRegistry codecRegistry, Class keyType,
+ Class valueType, TableCache.CACHETYPE cachetype) {
this.rawTable = rawTable;
this.codecRegistry = codecRegistry;
this.keyType = keyType;
this.valueType = valueType;
+if (cachetype == TableCache.CACHETYPE.FULLCACHE) {
Review comment:
Thanks, Anu for the comment, removed the cache type.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242868)
Time Spent: 7h 10m (was: 7h)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242867=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242867 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 15/May/19 21:24 Start Date: 15/May/19 21:24 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r284457204 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStore.java ## @@ -44,17 +45,20 @@ */ Table getTable(String name) throws IOException; + /** * Gets an existing TableStore with implicit key/value conversion. * * @param name - Name of the TableStore to get * @param keyType * @param valueType + * @param cachetype - Type of cache need to be used for this table. * @return - TableStore. * @throws IOException on Failure */ Table getTable(String name, - Class keyType, Class valueType) throws IOException; + Class keyType, Class valueType, + TableCache.CACHETYPE cachetype) throws IOException; Review comment: Removed the cache type. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 242867) Time Spent: 7h (was: 6h 50m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 7h > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242819=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242819 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 15/May/19 19:56 Start Date: 15/May/19 19:56 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-492800469 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 34 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 27 | Maven dependency ordering for branch | | +1 | mvninstall | 392 | trunk passed | | +1 | compile | 201 | trunk passed | | +1 | checkstyle | 52 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 810 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 132 | trunk passed | | 0 | spotbugs | 234 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 415 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 27 | Maven dependency ordering for patch | | +1 | mvninstall | 388 | the patch passed | | +1 | compile | 206 | the patch passed | | +1 | javac | 206 | the patch passed | | +1 | checkstyle | 61 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 664 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 128 | the patch passed | | +1 | findbugs | 435 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 147 | hadoop-hdds in the patch failed. | | -1 | unit | 846 | hadoop-ozone in the patch failed. | | +1 | asflicense | 40 | The patch does not generate ASF License warnings. | | | | 5192 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-798/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/798 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux cba0149be2ba 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 9569015 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/6/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/6/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/6/testReport/ | | Max. process+thread count | 5138 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/server-scm hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/6/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 242819) Time Spent: 6h 50m (was: 6h 40m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 6h 50m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242729=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242729 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 15/May/19 18:29 Start Date: 15/May/19 18:29 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-492770431 Thank You @arp7 for the review. Few of the review comments I have addressed them, for other's provided reply for the comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 242729) Time Spent: 6h 40m (was: 6.5h) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 6h 40m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242727=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242727
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 15/May/19 18:28
Start Date: 15/May/19 18:28
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284392124
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -69,8 +94,40 @@ public boolean isEmpty() throws IOException {
return rawTable.isEmpty();
}
+ /**
+ * Returns the value mapped to the given key in byte array or returns null
+ * if the key is not found.
+ *
+ * First it will check from cache, if it has entry return the value
+ * otherwise, get from the RocksDB table.
+ *
+ * @param key metadata key
+ * @return VALUE
+ * @throws IOException
+ */
@Override
public VALUE get(KEY key) throws IOException {
+// Here the metadata lock will guarantee that cache is not updated for same
+// key during get key.
+if (cache != null) {
+ CacheValue cacheValue = cache.get(new CacheKey<>(key));
+ if (cacheValue == null) {
+return getFromTable(key);
+ } else {
+// Doing this because, if the Cache Value Last operation is deleted
+// means it will eventually removed from DB. So, we should return null.
+if (cacheValue.getLastOperation() != CacheValue.OperationType.DELETED)
{
+ return cacheValue.getValue();
+} else {
+ return null;
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242727)
Time Spent: 6.5h (was: 6h 20m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242692
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 15/May/19 17:22
Start Date: 15/May/19 17:22
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284365121
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+/**
+ * Cache used for RocksDB tables.
+ * @param
+ * @param
+ */
+
+@Private
+@Evolving
+public interface TableCache {
+
+ /**
+ * Return the value for the key if it is present, otherwise return null.
+ * @param cacheKey
+ * @return CACHEVALUE
+ */
+ CACHEVALUE get(CACHEKEY cacheKey);
+
+ /**
+ * Add an entry to the cache, if the key already exists it overrides.
+ * @param cacheKey
+ * @param value
+ */
+ void put(CACHEKEY cacheKey, CACHEVALUE value);
+
+ /**
+ * Removes all the entries from the cache which are having epoch value less
+ * than or equal to specified epoch value.
+ * @param epoch
+ */
+ void cleanup(long epoch);
+
+ /**
+ * Return the size of the cache.
+ * @return size
+ */
+ int size();
+
+ /**
+ * Defines type of cache need to be used by OM RocksDB tables.
+ */
+ enum CACHETYPE {
Review comment:
The logic for get() actually takes care of that, I feel for bucket/volume
table having full cache will be useful as this is used for all the request
validation's. So, if we have in cache this will be a quick operation instead of
disk access.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242692)
Time Spent: 6h 20m (was: 6h 10m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242690=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242690
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 15/May/19 17:19
Start Date: 15/May/19 17:19
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284364154
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
##
@@ -245,42 +247,50 @@ protected DBStoreBuilder
addOMTablesAndCodecs(DBStoreBuilder builder) {
*/
protected void initializeOmTables() throws IOException {
userTable =
-this.store.getTable(USER_TABLE, String.class, VolumeList.class);
+this.store.getTable(USER_TABLE, String.class, VolumeList.class,
Review comment:
As for some tables where SCM does not need to have cache, so that is the
reason for introducing CacheType.
And also for some tables like deletedtable, where this is used by background
thread inOM, these does not need cache.
And as for volume/bucket table we plan to have entire table info, and
cleanup for them is no-op, even after flush to db, where as for partial cache
the cache entries will be cleaned up. This is the only difference between full
table cache vs partial table cache.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242690)
Time Spent: 6h 10m (was: 6h)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242684=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242684
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 15/May/19 17:18
Start Date: 15/May/19 17:18
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284363840
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+/**
+ * Cache used for RocksDB tables.
+ * @param
+ * @param
+ */
+
+@Private
+@Evolving
+public interface TableCache {
+
+ /**
+ * Return the value for the key if it is present, otherwise return null.
+ * @param cacheKey
+ * @return CACHEVALUE
+ */
+ CACHEVALUE get(CACHEKEY cacheKey);
+
+ /**
+ * Add an entry to the cache, if the key already exists it overrides.
+ * @param cacheKey
+ * @param value
+ */
+ void put(CACHEKEY cacheKey, CACHEVALUE value);
+
+ /**
+ * Removes all the entries from the cache which are having epoch value less
+ * than or equal to specified epoch value.
+ * @param epoch
+ */
+ void cleanup(long epoch);
+
+ /**
+ * Return the size of the cache.
+ * @return size
+ */
+ int size();
+
+ /**
+ * Defines type of cache need to be used by OM RocksDB tables.
+ */
+ enum CACHETYPE {
Review comment:
Okay let's skip the full cache then for now. We can add it later as an
optimization. All caches will be partial, and if an entry is not found in the
cache we always go to RocksDB. It should simplify the patch a bit. What do you
think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242684)
Time Spent: 6h (was: 5h 50m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242682=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242682
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 15/May/19 17:17
Start Date: 15/May/19 17:17
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284363194
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+/**
+ * Cache used for RocksDB tables.
+ * @param
+ * @param
+ */
+
+@Private
+@Evolving
+public interface TableCache {
+
+ /**
+ * Return the value for the key if it is present, otherwise return null.
+ * @param cacheKey
+ * @return CACHEVALUE
+ */
+ CACHEVALUE get(CACHEKEY cacheKey);
+
+ /**
+ * Add an entry to the cache, if the key already exists it overrides.
+ * @param cacheKey
+ * @param value
+ */
+ void put(CACHEKEY cacheKey, CACHEVALUE value);
+
+ /**
+ * Removes all the entries from the cache which are having epoch value less
+ * than or equal to specified epoch value.
+ * @param epoch
+ */
+ void cleanup(long epoch);
+
+ /**
+ * Return the size of the cache.
+ * @return size
+ */
+ int size();
+
+ /**
+ * Defines type of cache need to be used by OM RocksDB tables.
+ */
+ enum CACHETYPE {
Review comment:
Actually, this cache is used for correctness reasons for further operations
validation and read operation correctness. (As when double buffer comes in to
the place we don't commit to DB immediately)
And also we are cleaning up the cache after flush, so I am not sure what is
the use of loading entries. This cache is mainly for correctness reasons.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242682)
Time Spent: 5h 50m (was: 5h 40m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242679=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242679
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 15/May/19 17:15
Start Date: 15/May/19 17:15
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284362406
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/FullTableCache.java
##
@@ -0,0 +1,67 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * This is the full table cache, where it uses concurrentHashMap internally,
+ * and does not do any evict or cleanup. This full table cache need to be
+ * used by tables where we want to cache the entire table with out any
+ * cleanup to the cache
+ * @param
+ * @param
+ */
+
+@Private
+@Evolving
+public class FullTableCache
+implements TableCache {
+
+ private final ConcurrentHashMap cache;
+
+ public FullTableCache() {
+cache = new ConcurrentHashMap<>();
+ }
+
+ @Override
+ public CACHEVALUE get(CACHEKEY cacheKey) {
+return cache.get(cacheKey);
+ }
+
+
+ @Override
+ public void put(CACHEKEY cacheKey, CACHEVALUE value) {
+cache.put(cacheKey, value);
+ }
+
+ @Override
+ public void cleanup(long epoch) {
+// Do nothing
Review comment:
As for full table cache, we never want to cleanup, it is a do nothing
operation.
Added to the java doc.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242679)
Time Spent: 5.5h (was: 5h 20m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242680=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242680
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 15/May/19 17:15
Start Date: 15/May/19 17:15
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284362457
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -69,8 +94,40 @@ public boolean isEmpty() throws IOException {
return rawTable.isEmpty();
}
+ /**
+ * Returns the value mapped to the given key in byte array or returns null
+ * if the key is not found.
+ *
+ * First it will check from cache, if it has entry return the value
+ * otherwise, get from the RocksDB table.
+ *
+ * @param key metadata key
+ * @return VALUE
+ * @throws IOException
+ */
@Override
public VALUE get(KEY key) throws IOException {
+// Here the metadata lock will guarantee that cache is not updated for same
+// key during get key.
+if (cache != null) {
+ CacheValue cacheValue = cache.get(new CacheKey<>(key));
+ if (cacheValue == null) {
+return getFromTable(key);
+ } else {
+// Doing this because, if the Cache Value Last operation is deleted
+// means it will eventually removed from DB. So, we should return null.
+if (cacheValue.getLastOperation() != CacheValue.OperationType.DELETED)
{
+ return cacheValue.getValue();
+} else {
+ return null;
Review comment:
Thanks. Also instead of null we can pass in Optional.absent(), same thing
but makes it very clear that the value can be missing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242680)
Time Spent: 5h 40m (was: 5.5h)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242678=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242678
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 15/May/19 17:14
Start Date: 15/May/19 17:14
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284361950
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/CacheValue.java
##
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import java.util.Objects;
+
+/**
+ * CacheValue for the RocksDB Table.
+ * @param
+ */
+public class CacheValue {
+
+ private VALUE value;
+ private OperationType lastOperation;
+ // This value is used for evict entries from cache.
+ // This value is set with ratis transaction context log entry index.
+ private long epoch;
+
+ public CacheValue(VALUE value, OperationType lastOperation, long epoch) {
+Objects.requireNonNull(value, "Value Should not be null in CacheValue");
+this.value = value;
+this.lastOperation = lastOperation;
+this.epoch = epoch;
+ }
+
+ public VALUE getValue() {
+return value;
+ }
+
+ public OperationType getLastOperation() {
+return lastOperation;
+ }
+
+ public long getEpoch() {
+return epoch;
+ }
+
+ /**
+ * Last happened Operation.
Review comment:
Removed this lastOperation field.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242678)
Time Spent: 5h 20m (was: 5h 10m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242676=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242676
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 15/May/19 17:13
Start Date: 15/May/19 17:13
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284361801
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -69,8 +94,40 @@ public boolean isEmpty() throws IOException {
return rawTable.isEmpty();
}
+ /**
+ * Returns the value mapped to the given key in byte array or returns null
+ * if the key is not found.
+ *
+ * First it will check from cache, if it has entry return the value
+ * otherwise, get from the RocksDB table.
+ *
+ * @param key metadata key
+ * @return VALUE
+ * @throws IOException
+ */
@Override
public VALUE get(KEY key) throws IOException {
+// Here the metadata lock will guarantee that cache is not updated for same
+// key during get key.
+if (cache != null) {
+ CacheValue cacheValue = cache.get(new CacheKey<>(key));
+ if (cacheValue == null) {
+return getFromTable(key);
+ } else {
+// Doing this because, if the Cache Value Last operation is deleted
+// means it will eventually removed from DB. So, we should return null.
+if (cacheValue.getLastOperation() != CacheValue.OperationType.DELETED)
{
+ return cacheValue.getValue();
+} else {
+ return null;
Review comment:
Yes, we can pass null value to CacheValue actual value. This will work.
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242676)
Time Spent: 5h 10m (was: 5h)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242649
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 15/May/19 16:44
Start Date: 15/May/19 16:44
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284350525
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -69,8 +94,40 @@ public boolean isEmpty() throws IOException {
return rawTable.isEmpty();
}
+ /**
+ * Returns the value mapped to the given key in byte array or returns null
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242649)
Time Spent: 5h (was: 4h 50m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242144=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242144
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 14/May/19 23:45
Start Date: 14/May/19 23:45
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284026725
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+/**
+ * Cache used for RocksDB tables.
+ * @param
+ * @param
+ */
+
+@Private
+@Evolving
+public interface TableCache {
+
+ /**
+ * Return the value for the key if it is present, otherwise return null.
+ * @param cacheKey
+ * @return CACHEVALUE
+ */
+ CACHEVALUE get(CACHEKEY cacheKey);
+
+ /**
+ * Add an entry to the cache, if the key already exists it overrides.
+ * @param cacheKey
+ * @param value
+ */
+ void put(CACHEKEY cacheKey, CACHEVALUE value);
+
+ /**
+ * Removes all the entries from the cache which are having epoch value less
+ * than or equal to specified epoch value.
+ * @param epoch
+ */
+ void cleanup(long epoch);
+
+ /**
+ * Return the size of the cache.
+ * @return size
+ */
+ int size();
+
+ /**
+ * Defines type of cache need to be used by OM RocksDB tables.
+ */
+ enum CACHETYPE {
Review comment:
One feedback from Anu was to not differentiate between full and partial
cache while initializing so caller does not have to make a choice.
Instead we try to limit cache size by configuration and load as many entries
as possible. If we cannot load all entries then we could set a flag that it is
a partial cache. The implementation will certainly be a bit more complex.
~~I don't have a strong opinion either way. I am +1 with current approach
also.~~
On second thought I am leaning towards removing this distinction.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242144)
Time Spent: 4h 50m (was: 4h 40m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242133=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242133
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 14/May/19 23:25
Start Date: 14/May/19 23:25
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284037057
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -69,8 +94,40 @@ public boolean isEmpty() throws IOException {
return rawTable.isEmpty();
}
+ /**
+ * Returns the value mapped to the given key in byte array or returns null
+ * if the key is not found.
+ *
+ * First it will check from cache, if it has entry return the value
+ * otherwise, get from the RocksDB table.
+ *
+ * @param key metadata key
+ * @return VALUE
+ * @throws IOException
+ */
@Override
public VALUE get(KEY key) throws IOException {
+// Here the metadata lock will guarantee that cache is not updated for same
+// key during get key.
+if (cache != null) {
+ CacheValue cacheValue = cache.get(new CacheKey<>(key));
+ if (cacheValue == null) {
+return getFromTable(key);
+ } else {
+// Doing this because, if the Cache Value Last operation is deleted
+// means it will eventually removed from DB. So, we should return null.
+if (cacheValue.getLastOperation() != CacheValue.OperationType.DELETED)
{
+ return cacheValue.getValue();
+} else {
+ return null;
Review comment:
Instead of making delete operation a special case, in case of delete we can
just push a null value into the cache. So it will automatically return null.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242133)
Time Spent: 4.5h (was: 4h 20m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242132=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242132
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 14/May/19 23:25
Start Date: 14/May/19 23:25
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284025787
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/CacheValue.java
##
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import java.util.Objects;
+
+/**
+ * CacheValue for the RocksDB Table.
+ * @param
+ */
+public class CacheValue {
+
+ private VALUE value;
+ private OperationType lastOperation;
+ // This value is used for evict entries from cache.
+ // This value is set with ratis transaction context log entry index.
+ private long epoch;
+
+ public CacheValue(VALUE value, OperationType lastOperation, long epoch) {
+Objects.requireNonNull(value, "Value Should not be null in CacheValue");
+this.value = value;
+this.lastOperation = lastOperation;
+this.epoch = epoch;
+ }
+
+ public VALUE getValue() {
+return value;
+ }
+
+ public OperationType getLastOperation() {
+return lastOperation;
+ }
+
+ public long getEpoch() {
+return epoch;
+ }
+
+ /**
+ * Last happened Operation.
Review comment:
Bharat, what if we support further operation types in future? I was thinking
whether we really need this _lastOperation_ field.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242132)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242135=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242135
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 14/May/19 23:25
Start Date: 14/May/19 23:25
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284036830
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -69,8 +94,40 @@ public boolean isEmpty() throws IOException {
return rawTable.isEmpty();
}
+ /**
+ * Returns the value mapped to the given key in byte array or returns null
Review comment:
Can you also add a comment that the caller is responsible for
synchronization.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242135)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242131=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242131
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 14/May/19 23:25
Start Date: 14/May/19 23:25
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284036222
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
##
@@ -245,42 +247,50 @@ protected DBStoreBuilder
addOMTablesAndCodecs(DBStoreBuilder builder) {
*/
protected void initializeOmTables() throws IOException {
userTable =
-this.store.getTable(USER_TABLE, String.class, VolumeList.class);
+this.store.getTable(USER_TABLE, String.class, VolumeList.class,
Review comment:
This is one argument for why we should not have partial/full distinction.
It makes initialization of the tables a little clunky.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242131)
Time Spent: 4h 20m (was: 4h 10m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242134=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242134 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 14/May/19 23:25 Start Date: 14/May/19 23:25 Worklog Time Spent: 10m Work Description: arp7 commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r284033851 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/FullTableCache.java ## @@ -0,0 +1,67 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.utils.db.cache; + +import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.classification.InterfaceStability.Evolving; + +import java.util.concurrent.ConcurrentHashMap; + +/** + * This is the full table cache, where it uses concurrentHashMap internally, + * and does not do any evict or cleanup. This full table cache need to be + * used by tables where we want to cache the entire table with out any + * cleanup to the cache + * @param + * @param + */ + +@Private +@Evolving +public class FullTableCache OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 4.5h > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242130=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242130
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 14/May/19 23:25
Start Date: 14/May/19 23:25
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284026725
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+/**
+ * Cache used for RocksDB tables.
+ * @param
+ * @param
+ */
+
+@Private
+@Evolving
+public interface TableCache {
+
+ /**
+ * Return the value for the key if it is present, otherwise return null.
+ * @param cacheKey
+ * @return CACHEVALUE
+ */
+ CACHEVALUE get(CACHEKEY cacheKey);
+
+ /**
+ * Add an entry to the cache, if the key already exists it overrides.
+ * @param cacheKey
+ * @param value
+ */
+ void put(CACHEKEY cacheKey, CACHEVALUE value);
+
+ /**
+ * Removes all the entries from the cache which are having epoch value less
+ * than or equal to specified epoch value.
+ * @param epoch
+ */
+ void cleanup(long epoch);
+
+ /**
+ * Return the size of the cache.
+ * @return size
+ */
+ int size();
+
+ /**
+ * Defines type of cache need to be used by OM RocksDB tables.
+ */
+ enum CACHETYPE {
Review comment:
One feedback from Anu was to not differentiate between full and partial
cache while initializing so caller does not have to make a choice.
Instead we try to limit cache size by configuration and load as many entries
as possible. If we cannot load all entries then we could set a flag that it is
a partial cache. The implementation will certainly be a bit more complex.
I don't have a strong opinion either way. I am +1 with current approach also.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242130)
Time Spent: 4h 20m (was: 4h 10m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=242136=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242136
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 14/May/19 23:25
Start Date: 14/May/19 23:25
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284034704
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/FullTableCache.java
##
@@ -0,0 +1,67 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * This is the full table cache, where it uses concurrentHashMap internally,
+ * and does not do any evict or cleanup. This full table cache need to be
+ * used by tables where we want to cache the entire table with out any
+ * cleanup to the cache
+ * @param
+ * @param
+ */
+
+@Private
+@Evolving
+public class FullTableCache
+implements TableCache {
+
+ private final ConcurrentHashMap cache;
+
+ public FullTableCache() {
+cache = new ConcurrentHashMap<>();
+ }
+
+ @Override
+ public CACHEVALUE get(CACHEKEY cacheKey) {
+return cache.get(cacheKey);
+ }
+
+
+ @Override
+ public void put(CACHEKEY cacheKey, CACHEVALUE value) {
+cache.put(cacheKey, value);
+ }
+
+ @Override
+ public void cleanup(long epoch) {
+// Do nothing
Review comment:
Can you throw NotImplementedYetException here to make it clear that the
implementation is pending.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242136)
Time Spent: 4h 40m (was: 4.5h)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239892=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239892 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 09/May/19 18:52 Start Date: 09/May/19 18:52 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-491023563 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 34 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 64 | Maven dependency ordering for branch | | +1 | mvninstall | 415 | trunk passed | | +1 | compile | 205 | trunk passed | | +1 | checkstyle | 53 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 802 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 130 | trunk passed | | 0 | spotbugs | 237 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 417 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 26 | Maven dependency ordering for patch | | +1 | mvninstall | 396 | the patch passed | | +1 | compile | 204 | the patch passed | | +1 | javac | 204 | the patch passed | | +1 | checkstyle | 58 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 721 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 113 | the patch passed | | +1 | findbugs | 428 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 132 | hadoop-hdds in the patch failed. | | -1 | unit | 1136 | hadoop-ozone in the patch failed. | | +1 | asflicense | 32 | The patch does not generate ASF License warnings. | | | | 5514 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdds.scm.pipeline.TestRatisPipelineProvider | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-798/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/798 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 3f5f3b7f3979 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 90add05 | | Default Java | 1.8.0_191 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/5/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/5/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/5/testReport/ | | Max. process+thread count | 4521 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/server-scm hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/5/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 239892) Time Spent: 4h 10m (was: 4h) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 4h 10m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239815
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 09/May/19 17:21
Start Date: 09/May/19 17:21
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r282584180
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/PartialTableCache.java
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import java.util.Iterator;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+
+
+/**
+ * This is used for the tables where we don't want to cache entire table in
+ * in-memory.
+ */
+@Private
+@Evolving
+public class PartialTableCache
+implements TableCache{
+
+ private final ConcurrentHashMap cache;
+ private final TreeSet> epochEntries;
+ private ExecutorService executorService;
+
+
+
+ public PartialTableCache() {
+cache = new ConcurrentHashMap<>();
+epochEntries = new TreeSet>();
+// Created a singleThreadExecutor, so one cleanup will be running at a
+// time.
+executorService = Executors.newSingleThreadExecutor();
+ }
+
+ @Override
+ public CACHEVALUE get(CACHEKEY cachekey) {
+return cache.get(cachekey);
+ }
+
+ @Override
+ public void put(CACHEKEY cacheKey, CACHEVALUE value) {
+cache.put(cacheKey, value);
+CacheValue cacheValue = (CacheValue) cache.get(cacheKey);
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 239815)
Time Spent: 3h 50m (was: 3h 40m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239817=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239817 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 09/May/19 17:21 Start Date: 09/May/19 17:21 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-490992025 Thank You @hanishakoneru for the review. I have addressed review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 239817) Time Spent: 4h (was: 3h 50m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239613=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239613 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 09/May/19 06:32 Start Date: 09/May/19 06:32 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-490762355 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 29 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 60 | Maven dependency ordering for branch | | +1 | mvninstall | 393 | trunk passed | | +1 | compile | 199 | trunk passed | | +1 | checkstyle | 51 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 823 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 126 | trunk passed | | 0 | spotbugs | 236 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 409 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 27 | Maven dependency ordering for patch | | +1 | mvninstall | 411 | the patch passed | | +1 | compile | 207 | the patch passed | | +1 | javac | 207 | the patch passed | | +1 | checkstyle | 57 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 656 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 126 | the patch passed | | +1 | findbugs | 429 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 135 | hadoop-hdds in the patch failed. | | -1 | unit | 1247 | hadoop-ozone in the patch failed. | | +1 | asflicense | 32 | The patch does not generate ASF License warnings. | | | | 5576 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.web.client.TestBuckets | | | hadoop.ozone.TestMiniOzoneCluster | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-798/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/798 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 235d56cf079b 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 0c5fa2e | | Default Java | 1.8.0_191 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/4/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/4/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/4/testReport/ | | Max. process+thread count | 5055 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/server-scm hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/4/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 239613) Time Spent: 3h 40m (was: 3.5h) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 3h 40m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239589=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239589
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 09/May/19 04:57
Start Date: 09/May/19 04:57
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r282338372
##
File path:
hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestTypedRDBTableStore.java
##
@@ -236,4 +249,66 @@ public void forEachAndIterator() throws Exception {
}
}
}
+
+ @Test
+ public void testTypedTableWithCache() throws Exception {
+int iterCount = 10;
+try (Table testTable = createTypedTableWithCache(
+"Seven", TableCache.CACHETYPE.FULLCACHE)) {
+
+ for (int x = 0; x < iterCount; x++) {
+String key = Integer.toString(x);
+String value = Integer.toString(x);
+testTable.addCacheEntry(new CacheKey<>(key), new CacheValue<>(value,
+CacheValue.OperationType.CREATED, x));
+ }
+
+ // As we have added to cache, so get should return value even if it
+ // does not exist in DB.
+ for (int x = 0; x < iterCount; x++) {
+Assert.assertEquals(Integer.toString(1),
+testTable.get(Integer.toString(1)));
+ }
+
+}
+ }
+
+ @Test
+ public void testTypedTableWithCacheWithFewDeletedOperationType()
+ throws Exception {
+int iterCount = 10;
+try (Table testTable = createTypedTableWithCache(
+"Seven", TableCache.CACHETYPE.PARTIALCACHE)) {
+
+ for (int x = 0; x < iterCount; x++) {
+String key = Integer.toString(x);
+String value = Integer.toString(x);
+if (x % 2 == 0) {
+ testTable.addCacheEntry(new CacheKey<>(key),
+ new CacheValue<>(value,
+ CacheValue.OperationType.CREATED, x));
+} else {
+ testTable.addCacheEntry(new CacheKey<>(key), new CacheValue<>(value,
+ CacheValue.OperationType.DELETED, x));
+}
+ }
+
+ // As we have added to cache, so get should return value even if it
+ // does not exist in DB.
+ for (int x = 0; x < iterCount; x++) {
+if (x % 2 == 0) {
+ Assert.assertEquals(Integer.toString(x),
+ testTable.get(Integer.toString(x)));
+} else {
+ Assert.assertNull(testTable.get(Integer.toString(x)));
+}
+ }
+
+ testTable.cleanupCache(5);
+
+ GenericTestUtils.waitFor(() ->
+ ((TypedTable) testTable).getCache().size() == 4,
+ 100, 5000);
+}
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 239589)
Time Spent: 3.5h (was: 3h 20m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239588=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239588 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 09/May/19 04:54 Start Date: 09/May/19 04:54 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r282337940 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/Table.java ## @@ -60,6 +62,9 @@ void putWithBatch(BatchOperation batch, KEY key, VALUE value) * Returns the value mapped to the given key in byte array or returns null * if the key is not found. * + * First it will check from cache, if it has entry return the value + * otherwise, get from the RocksDB table. + * Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 239588) Time Spent: 3h 20m (was: 3h 10m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239586=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239586
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 09/May/19 04:49
Start Date: 09/May/19 04:49
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r282337376
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/PartialTableCache.java
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import java.util.Iterator;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+
+
+/**
+ * This is used for the tables where we don't want to cache entire table in
+ * in-memory.
+ */
+@Private
+@Evolving
+public class PartialTableCache
+implements TableCache{
+
+ private final ConcurrentHashMap cache;
+ private final TreeSet> epochEntries;
+ private ExecutorService executorService;
+
+
+
+ public PartialTableCache() {
+cache = new ConcurrentHashMap<>();
+epochEntries = new TreeSet>();
+// Created a singleThreadExecutor, so one cleanup will be running at a
+// time.
+executorService = Executors.newSingleThreadExecutor();
+ }
+
+ @Override
+ public CACHEVALUE get(CACHEKEY cachekey) {
+return cache.get(cachekey);
+ }
+
+ @Override
+ public void put(CACHEKEY cacheKey, CACHEVALUE value) {
+cache.put(cacheKey, value);
+CacheValue cacheValue = (CacheValue) cache.get(cacheKey);
+epochEntries.add(new EpochEntry<>(cacheValue.getEpoch(), cacheKey));
+ }
+
+ @Override
+ public void cleanup(long epoch) {
+executorService.submit(() -> evictCache(epoch));
+ }
+
+ @Override
+ public int size() {
+return cache.size();
+ }
+
+ private void evictCache(long epoch) {
+EpochEntry currentEntry = null;
+for (Iterator iterator = epochEntries.iterator(); iterator.hasNext();) {
+ currentEntry = (EpochEntry) iterator.next();
+ CACHEKEY cachekey = currentEntry.getCachekey();
+ CacheValue cacheValue = (CacheValue) cache.get(cachekey);
+ if (cacheValue.getEpoch() <= epoch) {
+cache.remove(cachekey);
+iterator.remove();
+ }
+
+ // If currentEntry epoch is greater than epoch, we have deleted all
+ // entries less than specified epoch. So, we can break.
+ if (currentEntry.getEpoch() > epoch) {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 239586)
Time Spent: 3h 10m (was: 3h)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239584=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239584
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 09/May/19 04:47
Start Date: 09/May/19 04:47
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r282337065
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/PartialTableCache.java
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import java.util.Iterator;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+
+
+/**
+ * This is used for the tables where we don't want to cache entire table in
+ * in-memory.
+ */
+@Private
+@Evolving
+public class PartialTableCache
+implements TableCache{
Review comment:
The cache is maintained for correctness purpose for reads and validation of
subsequent requests. The cache will be cleared once we flush to DB.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 239584)
Time Spent: 3h (was: 2h 50m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239582=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239582
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 09/May/19 04:45
Start Date: 09/May/19 04:45
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r282336866
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -71,6 +96,27 @@ public boolean isEmpty() throws IOException {
@Override
public VALUE get(KEY key) throws IOException {
+// Here the metadata lock will guarantee that cache is not updated for same
+// key during get key.
+if (cache != null) {
+ CacheValue cacheValue = cache.get(new CacheKey<>(key));
+ if (cacheValue == null) {
+return getFromTable(key);
+ } else {
+// Doing this because, if the Cache Value Last operation is deleted
+// means it will eventually removed from DB. So, we should return null.
+if (cacheValue.getLastOperation() != CacheValue.OperationType.DELETED)
{
+ return cacheValue.getValue();
+} else {
+ return null;
+}
+ }
+} else {
+ return getFromTable(key);
Review comment:
For tables where the cache is disabled, we need to do as before just read
from DB and return data.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 239582)
Time Spent: 2h 50m (was: 2h 40m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239581=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239581 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 09/May/19 04:44 Start Date: 09/May/19 04:44 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r282336707 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/Table.java ## @@ -97,6 +102,28 @@ void putWithBatch(BatchOperation batch, KEY key, VALUE value) */ String getName() throws IOException; + /** + * Add entry to the table cache. + * + * If the cacheKey already exists, it will override the entry. + * @param cacheKey + * @param cacheValue + */ Review comment: Once after the operation is executed in applyTransaction just before releasing the lock and sending a response to the client we need to add the response into cache. So that next subsequent read/write requests validation can be done with cache/db data. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 239581) Time Spent: 2h 40m (was: 2.5h) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239580=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239580 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 09/May/19 04:42 Start Date: 09/May/19 04:42 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r282336434 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStore.java ## @@ -44,17 +45,20 @@ */ Table getTable(String name) throws IOException; + /** * Gets an existing TableStore with implicit key/value conversion. * * @param name - Name of the TableStore to get * @param keyType * @param valueType + * @param cachetype - Type of cache need to be used for this table. * @return - TableStore. * @throws IOException on Failure */ Table getTable(String name, - Class keyType, Class valueType) throws IOException; + Class keyType, Class valueType, + TableCache.CACHETYPE cachetype) throws IOException; Review comment: Added this because for a few tables like bucket and volume table plan is to maintain full table information, for other tables we maintain a partial cache, whereas for few tables we don't want to maintain cache at all. (This is a common interface for all tables in Ozone SCM/OM. So, having this option will help to know which kind of cache need to be used for the table.) As these are frequently used for validation of almost every operation in OM. So, this might improve validation like bucket/volume exists or not checks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 239580) Time Spent: 2.5h (was: 2h 20m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239520=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239520
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 08/May/19 23:54
Start Date: 08/May/19 23:54
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r282297165
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/PartialTableCache.java
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import java.util.Iterator;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+
+
+/**
+ * This is used for the tables where we don't want to cache entire table in
+ * in-memory.
+ */
+@Private
+@Evolving
+public class PartialTableCache
+implements TableCache{
+
+ private final ConcurrentHashMap cache;
+ private final TreeSet> epochEntries;
+ private ExecutorService executorService;
+
+
+
+ public PartialTableCache() {
+cache = new ConcurrentHashMap<>();
+epochEntries = new TreeSet>();
+// Created a singleThreadExecutor, so one cleanup will be running at a
+// time.
+executorService = Executors.newSingleThreadExecutor();
+ }
+
+ @Override
+ public CACHEVALUE get(CACHEKEY cachekey) {
+return cache.get(cachekey);
+ }
+
+ @Override
+ public void put(CACHEKEY cacheKey, CACHEVALUE value) {
+cache.put(cacheKey, value);
+CacheValue cacheValue = (CacheValue) cache.get(cacheKey);
Review comment:
Instead of casting the cache.get() object to CacheValue, I think CACHEVALUE
itself should extend CacheValue so that it is guaranteed that the Value part of
TableCache is an instance of CacheValue.class. Same for CACHEKEY also.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 239520)
Time Spent: 2h 10m (was: 2h)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239519=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239519
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 08/May/19 23:54
Start Date: 08/May/19 23:54
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r282214231
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/PartialTableCache.java
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import java.util.Iterator;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+
+
+/**
+ * This is used for the tables where we don't want to cache entire table in
+ * in-memory.
+ */
+@Private
+@Evolving
+public class PartialTableCache
+implements TableCache{
+
+ private final ConcurrentHashMap cache;
+ private final TreeSet> epochEntries;
+ private ExecutorService executorService;
+
+
+
+ public PartialTableCache() {
+cache = new ConcurrentHashMap<>();
+epochEntries = new TreeSet>();
+// Created a singleThreadExecutor, so one cleanup will be running at a
+// time.
+executorService = Executors.newSingleThreadExecutor();
+ }
+
+ @Override
+ public CACHEVALUE get(CACHEKEY cachekey) {
+return cache.get(cachekey);
+ }
+
+ @Override
+ public void put(CACHEKEY cacheKey, CACHEVALUE value) {
+cache.put(cacheKey, value);
+CacheValue cacheValue = (CacheValue) cache.get(cacheKey);
+epochEntries.add(new EpochEntry<>(cacheValue.getEpoch(), cacheKey));
+ }
+
+ @Override
+ public void cleanup(long epoch) {
+executorService.submit(() -> evictCache(epoch));
+ }
+
+ @Override
+ public int size() {
+return cache.size();
+ }
+
+ private void evictCache(long epoch) {
+EpochEntry currentEntry = null;
+for (Iterator iterator = epochEntries.iterator(); iterator.hasNext();) {
+ currentEntry = (EpochEntry) iterator.next();
+ CACHEKEY cachekey = currentEntry.getCachekey();
+ CacheValue cacheValue = (CacheValue) cache.get(cachekey);
+ if (cacheValue.getEpoch() <= epoch) {
+cache.remove(cachekey);
+iterator.remove();
+ }
+
+ // If currentEntry epoch is greater than epoch, we have deleted all
+ // entries less than specified epoch. So, we can break.
+ if (currentEntry.getEpoch() > epoch) {
Review comment:
We can avoid the 2nd if check and put it in else block.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 239519)
Time Spent: 2h (was: 1h 50m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239518=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239518
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 08/May/19 23:54
Start Date: 08/May/19 23:54
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #798:
HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r282215862
##
File path:
hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestTypedRDBTableStore.java
##
@@ -236,4 +249,66 @@ public void forEachAndIterator() throws Exception {
}
}
}
+
+ @Test
+ public void testTypedTableWithCache() throws Exception {
+int iterCount = 10;
+try (Table testTable = createTypedTableWithCache(
+"Seven", TableCache.CACHETYPE.FULLCACHE)) {
+
+ for (int x = 0; x < iterCount; x++) {
+String key = Integer.toString(x);
+String value = Integer.toString(x);
+testTable.addCacheEntry(new CacheKey<>(key), new CacheValue<>(value,
+CacheValue.OperationType.CREATED, x));
+ }
+
+ // As we have added to cache, so get should return value even if it
+ // does not exist in DB.
+ for (int x = 0; x < iterCount; x++) {
+Assert.assertEquals(Integer.toString(1),
+testTable.get(Integer.toString(1)));
+ }
+
+}
+ }
+
+ @Test
+ public void testTypedTableWithCacheWithFewDeletedOperationType()
+ throws Exception {
+int iterCount = 10;
+try (Table testTable = createTypedTableWithCache(
+"Seven", TableCache.CACHETYPE.PARTIALCACHE)) {
+
+ for (int x = 0; x < iterCount; x++) {
+String key = Integer.toString(x);
+String value = Integer.toString(x);
+if (x % 2 == 0) {
+ testTable.addCacheEntry(new CacheKey<>(key),
+ new CacheValue<>(value,
+ CacheValue.OperationType.CREATED, x));
+} else {
+ testTable.addCacheEntry(new CacheKey<>(key), new CacheValue<>(value,
+ CacheValue.OperationType.DELETED, x));
+}
+ }
+
+ // As we have added to cache, so get should return value even if it
+ // does not exist in DB.
+ for (int x = 0; x < iterCount; x++) {
+if (x % 2 == 0) {
+ Assert.assertEquals(Integer.toString(x),
+ testTable.get(Integer.toString(x)));
+} else {
+ Assert.assertNull(testTable.get(Integer.toString(x)));
+}
+ }
+
+ testTable.cleanupCache(5);
+
+ GenericTestUtils.waitFor(() ->
+ ((TypedTable) testTable).getCache().size() == 4,
+ 100, 5000);
+}
Review comment:
Can we also check that the cache entries remaining in the cache are the
expected entries.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 239518)
Time Spent: 1h 50m (was: 1h 40m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239521=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239521 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 08/May/19 23:54 Start Date: 08/May/19 23:54 Worklog Time Spent: 10m Work Description: hanishakoneru commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r282189907 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/Table.java ## @@ -60,6 +62,9 @@ void putWithBatch(BatchOperation batch, KEY key, VALUE value) * Returns the value mapped to the given key in byte array or returns null * if the key is not found. * + * First it will check from cache, if it has entry return the value + * otherwise, get from the RocksDB table. + * Review comment: The RDBTable implementation of Table does not check the cache. We should probably move this statement to TypedTable which implements the cache. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 239521) Time Spent: 2h 20m (was: 2h 10m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239331=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239331
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 08/May/19 17:30
Start Date: 08/May/19 17:30
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r282167175
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -71,6 +96,27 @@ public boolean isEmpty() throws IOException {
@Override
public VALUE get(KEY key) throws IOException {
+// Here the metadata lock will guarantee that cache is not updated for same
+// key during get key.
+if (cache != null) {
+ CacheValue cacheValue = cache.get(new CacheKey<>(key));
+ if (cacheValue == null) {
+return getFromTable(key);
+ } else {
+// Doing this because, if the Cache Value Last operation is deleted
+// means it will eventually removed from DB. So, we should return null.
+if (cacheValue.getLastOperation() != CacheValue.OperationType.DELETED)
{
Review comment:
Why do we even cache the deleted Operations? Delete is not in the
performance critical path at all. If you can instruct the system to make the
full commit or flush the buffer when there is a delete op you don't need to
keep this extra state in the cache. yes, repeated deletes will call state
machine call back. When do we actually flush / clear this entry?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 239331)
Time Spent: 1h 10m (was: 1h)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239332=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239332 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 08/May/19 17:30 Start Date: 08/May/19 17:30 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r282164867 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/Table.java ## @@ -97,6 +102,28 @@ void putWithBatch(BatchOperation batch, KEY key, VALUE value) */ String getName() throws IOException; + /** + * Add entry to the table cache. + * + * If the cacheKey already exists, it will override the entry. + * @param cacheKey + * @param cacheValue + */ Review comment: well, I was really hoping that the fact that there is a cache is not visible to the layer that is reading and writing. Is there a reason why that should be exposed to calling applications? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 239332) Time Spent: 1h 20m (was: 1h 10m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239330=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239330
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 08/May/19 17:30
Start Date: 08/May/19 17:30
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r282168861
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/PartialTableCache.java
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import java.util.Iterator;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+
+
+/**
+ * This is used for the tables where we don't want to cache entire table in
+ * in-memory.
+ */
+@Private
+@Evolving
+public class PartialTableCache
+implements TableCache{
Review comment:
Not sure if you have seen this,
https://github.com/facebook/rocksdb/wiki/Block-Cache
We already do this cache control in the RockDB. I am not sure if we should
do this twice. Unless you have a lookup problem which cannot be solved by
hashing or prefix lookup, we will have more efficient usage of memory by
relying on the underlying layer and more over having a unified cache layer will
lead to better cache layer utilization.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 239330)
Time Spent: 1h (was: 50m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239335=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239335 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 08/May/19 17:30 Start Date: 08/May/19 17:30 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r282164337 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStore.java ## @@ -44,17 +45,20 @@ */ Table getTable(String name) throws IOException; + /** * Gets an existing TableStore with implicit key/value conversion. * * @param name - Name of the TableStore to get * @param keyType * @param valueType + * @param cachetype - Type of cache need to be used for this table. * @return - TableStore. * @throws IOException on Failure */ Table getTable(String name, - Class keyType, Class valueType) throws IOException; + Class keyType, Class valueType, + TableCache.CACHETYPE cachetype) throws IOException; Review comment: Why do we need an external visible TableCache.CACHETYPE ? shouldn't this be an implementation detail of the Tables that have Cache? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 239335) Time Spent: 1h 40m (was: 1.5h) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239333=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239333
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 08/May/19 17:30
Start Date: 08/May/19 17:30
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r282165925
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -31,22 +38,40 @@
*/
public class TypedTable implements Table {
- private Table rawTable;
+ private final Table rawTable;
+
+ private final CodecRegistry codecRegistry;
- private CodecRegistry codecRegistry;
+ private final Class keyType;
- private Class keyType;
+ private final Class valueType;
- private Class valueType;
+ private final TableCache, CacheValue> cache;
public TypedTable(
Table rawTable,
CodecRegistry codecRegistry, Class keyType,
Class valueType) {
+this(rawTable, codecRegistry, keyType, valueType,
+null);
+ }
+
+
+ public TypedTable(
+ Table rawTable,
+ CodecRegistry codecRegistry, Class keyType,
+ Class valueType, TableCache.CACHETYPE cachetype) {
this.rawTable = rawTable;
this.codecRegistry = codecRegistry;
this.keyType = keyType;
this.valueType = valueType;
+if (cachetype == TableCache.CACHETYPE.FULLCACHE) {
Review comment:
It is impossible for the user to tell you apriori if they want a full cache
or partial cache. When you start a cluster you always want a full cache. We
should get a cache size -- or get a percentage of memory from the OM cache size
and use that if needed. Or for time being rely on the RocksDB doing the right
thing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 239333)
Time Spent: 1.5h (was: 1h 20m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239329=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239329
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 08/May/19 17:30
Start Date: 08/May/19 17:30
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r282169161
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/PartialTableCache.java
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import java.util.Iterator;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+
+
+/**
+ * This is used for the tables where we don't want to cache entire table in
+ * in-memory.
+ */
+@Private
+@Evolving
+public class PartialTableCache
+implements TableCache{
+
+ private final ConcurrentHashMap cache;
+ private final TreeSet> epochEntries;
+ private ExecutorService executorService;
+
+
+
+ public PartialTableCache() {
+cache = new ConcurrentHashMap<>();
+epochEntries = new TreeSet>();
+// Created a singleThreadExecutor, so one cleanup will be running at a
+// time.
+executorService = Executors.newSingleThreadExecutor();
+ }
+
+ @Override
+ public CACHEVALUE get(CACHEKEY cachekey) {
+return cache.get(cachekey);
+ }
+
+ @Override
+ public void put(CACHEKEY cacheKey, CACHEVALUE value) {
+cache.put(cacheKey, value);
+CacheValue cacheValue = (CacheValue) cache.get(cacheKey);
+epochEntries.add(new EpochEntry<>(cacheValue.getEpoch(), cacheKey));
+ }
+
+ @Override
+ public void cleanup(long epoch) {
+executorService.submit(() -> evictCache(epoch));
+ }
+
+ @Override
+ public int size() {
+return cache.size();
+ }
+
+ private void evictCache(long epoch) {
Review comment:
Shouldn't a key be evicted if it was a delete operation and the state
machine commit has taken place ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 239329)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[
https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=239328=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239328
]
ASF GitHub Bot logged work on HDDS-1499:
Author: ASF GitHub Bot
Created on: 08/May/19 17:30
Start Date: 08/May/19 17:30
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r282167434
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -71,6 +96,27 @@ public boolean isEmpty() throws IOException {
@Override
public VALUE get(KEY key) throws IOException {
+// Here the metadata lock will guarantee that cache is not updated for same
+// key during get key.
+if (cache != null) {
+ CacheValue cacheValue = cache.get(new CacheKey<>(key));
+ if (cacheValue == null) {
+return getFromTable(key);
+ } else {
+// Doing this because, if the Cache Value Last operation is deleted
+// means it will eventually removed from DB. So, we should return null.
+if (cacheValue.getLastOperation() != CacheValue.OperationType.DELETED)
{
+ return cacheValue.getValue();
+} else {
+ return null;
+}
+ }
+} else {
+ return getFromTable(key);
Review comment:
Not sure if you need this get again ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 239328)
Time Spent: 50m (was: 40m)
> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to
> flush transaction in a batch, instead of using rocksdb put() for every
> operation. When this comes in to place we need cache in OzoneManager HA to
> handle/server the requests for validation/returning responses.
>
> This Jira will implement Cache as an integral part of the table. In this way
> users using this table does not need to handle like check cache/db. For this,
> we can update get API in the table to handle the cache.
>
> This Jira will implement:
> # Cache as a part of each Table.
> # Uses this cache in get().
> # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=238843=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238843 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 07/May/19 22:00 Start Date: 07/May/19 22:00 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-490270992 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 41 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 22 | Maven dependency ordering for branch | | +1 | mvninstall | 398 | trunk passed | | +1 | compile | 201 | trunk passed | | +1 | checkstyle | 56 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 970 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 129 | trunk passed | | 0 | spotbugs | 241 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 432 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for patch | | +1 | mvninstall | 402 | the patch passed | | +1 | compile | 227 | the patch passed | | +1 | javac | 227 | the patch passed | | +1 | checkstyle | 65 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 778 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 131 | the patch passed | | +1 | findbugs | 478 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 167 | hadoop-hdds in the patch failed. | | -1 | unit | 1605 | hadoop-ozone in the patch failed. | | -1 | asflicense | 35 | The patch generated 1 ASF License warnings. | | | | 6340 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdds.scm.pipeline.TestRatisPipelineProvider | | | hadoop.hdds.scm.safemode.TestSCMSafeModeWithPipelineRules | | | hadoop.ozone.container.TestContainerReplication | | | hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient | | | hadoop.ozone.client.rpc.TestReadRetries | | | hadoop.ozone.client.rpc.TestContainerStateMachine | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-798/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/798 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux c340489b69ee 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 7f0e2c6 | | Default Java | 1.8.0_191 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/3/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/3/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/3/testReport/ | | asflicense | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/3/artifact/out/patch-asflicense-problems.txt | | Max. process+thread count | 4533 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/server-scm hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 238843) Time Spent: 40m (was: 0.5h) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL:
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=238741=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238741 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 07/May/19 19:24 Start Date: 07/May/19 19:24 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-49070 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 42 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 25 | Maven dependency ordering for branch | | +1 | mvninstall | 398 | trunk passed | | +1 | compile | 209 | trunk passed | | +1 | checkstyle | 53 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 863 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 120 | trunk passed | | 0 | spotbugs | 240 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 420 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for patch | | +1 | mvninstall | 406 | the patch passed | | +1 | compile | 205 | the patch passed | | +1 | javac | 205 | the patch passed | | -0 | checkstyle | 29 | hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 725 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 126 | the patch passed | | +1 | findbugs | 451 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 155 | hadoop-hdds in the patch failed. | | -1 | unit | 1197 | hadoop-ozone in the patch failed. | | +1 | asflicense | 32 | The patch does not generate ASF License warnings. | | | | 5651 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.web.client.TestKeysRatis | | | hadoop.hdds.scm.pipeline.TestSCMPipelineManager | | | hadoop.ozone.container.TestContainerReplication | | | hadoop.ozone.client.rpc.TestCommitWatcher | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-798/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/798 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux e291fdb1560b 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 1a696cc | | Default Java | 1.8.0_191 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/2/artifact/out/diff-checkstyle-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/2/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/2/testReport/ | | Max. process+thread count | 4612 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/server-scm hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 238741) Time Spent: 0.5h (was: 20m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=238720=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238720 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 07/May/19 19:04 Start Date: 07/May/19 19:04 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-490215309 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 29 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 63 | Maven dependency ordering for branch | | +1 | mvninstall | 405 | trunk passed | | +1 | compile | 195 | trunk passed | | +1 | checkstyle | 52 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 815 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 128 | trunk passed | | 0 | spotbugs | 235 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 411 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 28 | Maven dependency ordering for patch | | +1 | mvninstall | 404 | the patch passed | | +1 | compile | 210 | the patch passed | | +1 | javac | 210 | the patch passed | | -0 | checkstyle | 31 | hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 654 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 117 | the patch passed | | +1 | findbugs | 433 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 136 | hadoop-hdds in the patch failed. | | -1 | unit | 867 | hadoop-ozone in the patch failed. | | +1 | asflicense | 36 | The patch does not generate ASF License warnings. | | | | 5197 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-798/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/798 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux e994bc971635 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 1a696cc | | Default Java | 1.8.0_191 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/1/artifact/out/diff-checkstyle-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/1/testReport/ | | Max. process+thread count | 4725 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/server-scm hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 238720) Time Spent: 20m (was: 10m) > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels:
[jira] [Work logged] (HDDS-1499) OzoneManager Cache
[ https://issues.apache.org/jira/browse/HDDS-1499?focusedWorklogId=238679=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238679 ] ASF GitHub Bot logged work on HDDS-1499: Author: ASF GitHub Bot Created on: 07/May/19 17:36 Start Date: 07/May/19 17:36 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 238679) Time Spent: 10m Remaining Estimate: 0h > OzoneManager Cache > -- > > Key: HDDS-1499 > URL: https://issues.apache.org/jira/browse/HDDS-1499 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In this Jira, we shall implement a cache for Table. > As with OM HA, we are planning to implement double buffer implementation to > flush transaction in a batch, instead of using rocksdb put() for every > operation. When this comes in to place we need cache in OzoneManager HA to > handle/server the requests for validation/returning responses. > > This Jira will implement Cache as an integral part of the table. In this way > users using this table does not need to handle like check cache/db. For this, > we can update get API in the table to handle the cache. > > This Jira will implement: > # Cache as a part of each Table. > # Uses this cache in get(). > # Exposes api for cleanup, add entries to cache. > Usage to add the entries in to cache will be done in further jira's. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
