[jira] [Work logged] (HDDS-1696) RocksDB use separate Write-ahead-log location for RocksDB.
[ https://issues.apache.org/jira/browse/HDDS-1696?focusedWorklogId=266214=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266214 ] ASF GitHub Bot logged work on HDDS-1696: Author: ASF GitHub Bot Created on: 24/Jun/19 23:31 Start Date: 24/Jun/19 23:31 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #981: HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB. URL: https://github.com/apache/hadoop/pull/981 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266214) Time Spent: 2h (was: 1h 50m) > RocksDB use separate Write-ahead-log location for RocksDB. > -- > > Key: HDDS-1696 > URL: https://issues.apache.org/jira/browse/HDDS-1696 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > This will help on production systems where WAL logs and db actual data files > will be in a different location. During compaction, it will not affect actual > writes. > > Suggested by [~msingh] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1696) RocksDB use separate Write-ahead-log location for RocksDB.
[ https://issues.apache.org/jira/browse/HDDS-1696?focusedWorklogId=266213=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266213 ] ASF GitHub Bot logged work on HDDS-1696: Author: ASF GitHub Bot Created on: 24/Jun/19 23:31 Start Date: 24/Jun/19 23:31 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #981: HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB. URL: https://github.com/apache/hadoop/pull/981#issuecomment-505219413 Closing this, as there is a way to pass a .ini file and read the values and set the required rocksdb. Thank You @anuengineer and @arp7 for the info. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266213) Time Spent: 1h 50m (was: 1h 40m) > RocksDB use separate Write-ahead-log location for RocksDB. > -- > > Key: HDDS-1696 > URL: https://issues.apache.org/jira/browse/HDDS-1696 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > This will help on production systems where WAL logs and db actual data files > will be in a different location. During compaction, it will not affect actual > writes. > > Suggested by [~msingh] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1696) RocksDB use separate Write-ahead-log location for RocksDB.
[ https://issues.apache.org/jira/browse/HDDS-1696?focusedWorklogId=261889=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261889 ] ASF GitHub Bot logged work on HDDS-1696: Author: ASF GitHub Bot Created on: 17/Jun/19 22:51 Start Date: 17/Jun/19 22:51 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #981: HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB. URL: https://github.com/apache/hadoop/pull/981#discussion_r294548907 ## File path: hadoop-hdds/common/src/main/resources/ozone-default.xml ## @@ -646,6 +646,17 @@ production environments. + Review comment: https://cwiki.apache.org/confluence/display/HADOOP/Java-based+configuration+API This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 261889) Time Spent: 1h 40m (was: 1.5h) > RocksDB use separate Write-ahead-log location for RocksDB. > -- > > Key: HDDS-1696 > URL: https://issues.apache.org/jira/browse/HDDS-1696 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > This will help on production systems where WAL logs and db actual data files > will be in a different location. During compaction, it will not affect actual > writes. > > Suggested by [~msingh] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1696) RocksDB use separate Write-ahead-log location for RocksDB.
[ https://issues.apache.org/jira/browse/HDDS-1696?focusedWorklogId=261871=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261871 ] ASF GitHub Bot logged work on HDDS-1696: Author: ASF GitHub Bot Created on: 17/Jun/19 22:30 Start Date: 17/Jun/19 22:30 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #981: HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB. URL: https://github.com/apache/hadoop/pull/981#discussion_r294540670 ## File path: hadoop-hdds/common/src/main/resources/ozone-default.xml ## @@ -2327,6 +2349,17 @@ production environments. + +ozone.recon.db.wal.dir Review comment: same. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 261871) Time Spent: 40m (was: 0.5h) > RocksDB use separate Write-ahead-log location for RocksDB. > -- > > Key: HDDS-1696 > URL: https://issues.apache.org/jira/browse/HDDS-1696 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > This will help on production systems where WAL logs and db actual data files > will be in a different location. During compaction, it will not affect actual > writes. > > Suggested by [~msingh] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1696) RocksDB use separate Write-ahead-log location for RocksDB.
[
https://issues.apache.org/jira/browse/HDDS-1696?focusedWorklogId=261876=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261876
]
ASF GitHub Bot logged work on HDDS-1696:
Author: ASF GitHub Bot
Created on: 17/Jun/19 22:30
Start Date: 17/Jun/19 22:30
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #981: HDDS-1696.
RocksDB use separate Write-ahead-log location for OM RocksDB.
URL: https://github.com/apache/hadoop/pull/981#discussion_r294543691
##
File path:
hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestDBStoreBuilder.java
##
@@ -71,12 +71,26 @@ public void builderWithOneParamV2() throws IOException {
if(!newFolder.exists()) {
Assert.assertTrue(newFolder.mkdirs());
}
-thrown.expect(IOException.class);
DBStoreBuilder.newBuilder(conf)
.setPath(newFolder.toPath())
.build();
}
+ @Test
+ public void builderWithWalDirSet() throws IOException {
+Configuration conf = new Configuration();
+File newFolder = folder.newFolder();
+File walDir = folder.newFolder();
+if(!newFolder.exists()) {
+ Assert.assertTrue(newFolder.mkdirs());
+}
+DBStoreBuilder.newBuilder(conf)
+.setPath(newFolder.toPath())
Review comment:
This test illustrates my concern. We will add this feature for performance
reasons but we really have no data to show or prove this is the root cause of
perf issues. Neither can we test it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 261876)
Time Spent: 1.5h (was: 1h 20m)
> RocksDB use separate Write-ahead-log location for RocksDB.
> --
>
> Key: HDDS-1696
> URL: https://issues.apache.org/jira/browse/HDDS-1696
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> This will help on production systems where WAL logs and db actual data files
> will be in a different location. During compaction, it will not affect actual
> writes.
>
> Suggested by [~msingh]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1696) RocksDB use separate Write-ahead-log location for RocksDB.
[ https://issues.apache.org/jira/browse/HDDS-1696?focusedWorklogId=261870=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261870 ] ASF GitHub Bot logged work on HDDS-1696: Author: ASF GitHub Bot Created on: 17/Jun/19 22:30 Start Date: 17/Jun/19 22:30 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #981: HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB. URL: https://github.com/apache/hadoop/pull/981#discussion_r294543351 ## File path: hadoop-hdds/common/src/main/resources/ozone-default.xml ## @@ -646,6 +646,17 @@ production environments. + +ozone.om.db.wal.dir Review comment: Use new style Config options please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 261870) Time Spent: 0.5h (was: 20m) > RocksDB use separate Write-ahead-log location for RocksDB. > -- > > Key: HDDS-1696 > URL: https://issues.apache.org/jira/browse/HDDS-1696 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > This will help on production systems where WAL logs and db actual data files > will be in a different location. During compaction, it will not affect actual > writes. > > Suggested by [~msingh] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1696) RocksDB use separate Write-ahead-log location for RocksDB.
[
https://issues.apache.org/jira/browse/HDDS-1696?focusedWorklogId=261873=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261873
]
ASF GitHub Bot logged work on HDDS-1696:
Author: ASF GitHub Bot
Created on: 17/Jun/19 22:30
Start Date: 17/Jun/19 22:30
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #981: HDDS-1696.
RocksDB use separate Write-ahead-log location for OM RocksDB.
URL: https://github.com/apache/hadoop/pull/981#discussion_r294540419
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStoreBuilder.java
##
@@ -220,4 +227,15 @@ private File getDBFile() throws IOException {
return Paths.get(dbPath.toString(), dbname).toFile();
}
+ private File getWALFile() throws IOException {
+if (walPath == null) {
+ LOG.error("Write-ahead-log path is " +
Review comment:
Ok, in most cases this would not happen. No one would map a WAL to a
different disk. If it is mapped, then we can probably log that we detected it
is mapped. This is certainly not an error.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 261873)
Time Spent: 1h (was: 50m)
> RocksDB use separate Write-ahead-log location for RocksDB.
> --
>
> Key: HDDS-1696
> URL: https://issues.apache.org/jira/browse/HDDS-1696
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> This will help on production systems where WAL logs and db actual data files
> will be in a different location. During compaction, it will not affect actual
> writes.
>
> Suggested by [~msingh]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1696) RocksDB use separate Write-ahead-log location for RocksDB.
[ https://issues.apache.org/jira/browse/HDDS-1696?focusedWorklogId=261878=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261878 ] ASF GitHub Bot logged work on HDDS-1696: Author: ASF GitHub Bot Created on: 17/Jun/19 22:30 Start Date: 17/Jun/19 22:30 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #981: HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB. URL: https://github.com/apache/hadoop/pull/981#discussion_r294540555 ## File path: hadoop-hdds/common/src/main/resources/ozone-default.xml ## @@ -646,6 +646,17 @@ production environments. + Review comment: Can we please use the new config style? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 261878) > RocksDB use separate Write-ahead-log location for RocksDB. > -- > > Key: HDDS-1696 > URL: https://issues.apache.org/jira/browse/HDDS-1696 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > This will help on production systems where WAL logs and db actual data files > will be in a different location. During compaction, it will not affect actual > writes. > > Suggested by [~msingh] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1696) RocksDB use separate Write-ahead-log location for RocksDB.
[
https://issues.apache.org/jira/browse/HDDS-1696?focusedWorklogId=261879=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261879
]
ASF GitHub Bot logged work on HDDS-1696:
Author: ASF GitHub Bot
Created on: 17/Jun/19 22:30
Start Date: 17/Jun/19 22:30
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #981: HDDS-1696.
RocksDB use separate Write-ahead-log location for OM RocksDB.
URL: https://github.com/apache/hadoop/pull/981#discussion_r294543803
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneHAClusterImpl.java
##
@@ -208,6 +209,15 @@ void initializeConfiguration() throws IOException {
// Set metadata/DB dir base path
String metaDirPath = path + "/" + nodeId;
conf.set(OZONE_METADATA_DIRS, metaDirPath);
+
+// If wal Dir is set, as in OM HA setup all OM's will be on the
+// same node, wal directories will conflict with each other.
+// Append nodeId similar to metaDirPath.
+String walDir = conf.get(OMConfigKeys.OZONE_OM_DB_WAL_DIR);
Review comment:
New Configs please.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 261879)
> RocksDB use separate Write-ahead-log location for RocksDB.
> --
>
> Key: HDDS-1696
> URL: https://issues.apache.org/jira/browse/HDDS-1696
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> This will help on production systems where WAL logs and db actual data files
> will be in a different location. During compaction, it will not affect actual
> writes.
>
> Suggested by [~msingh]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1696) RocksDB use separate Write-ahead-log location for RocksDB.
[ https://issues.apache.org/jira/browse/HDDS-1696?focusedWorklogId=261872=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261872 ] ASF GitHub Bot logged work on HDDS-1696: Author: ASF GitHub Bot Created on: 17/Jun/19 22:30 Start Date: 17/Jun/19 22:30 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #981: HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB. URL: https://github.com/apache/hadoop/pull/981#discussion_r294540185 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java ## @@ -35,6 +35,7 @@ // metadata dir but in future we may support multiple for redundancy or // performance. public static final String OZONE_SCM_DB_DIRS = "ozone.scm.db.dirs"; + public static final String OZONE_SCM_DB_WAL_DIR = "ozone.scm.db.wal.dir"; Review comment: Do we know that WAL performance is being compromised today ? This adds a new config value, but are we sure about the benefit ? In other words, are we sure that ozone is being bottlenecked by RocksDB? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 261872) Time Spent: 50m (was: 40m) > RocksDB use separate Write-ahead-log location for RocksDB. > -- > > Key: HDDS-1696 > URL: https://issues.apache.org/jira/browse/HDDS-1696 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > This will help on production systems where WAL logs and db actual data files > will be in a different location. During compaction, it will not affect actual > writes. > > Suggested by [~msingh] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1696) RocksDB use separate Write-ahead-log location for RocksDB.
[
https://issues.apache.org/jira/browse/HDDS-1696?focusedWorklogId=261877=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261877
]
ASF GitHub Bot logged work on HDDS-1696:
Author: ASF GitHub Bot
Created on: 17/Jun/19 22:30
Start Date: 17/Jun/19 22:30
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #981: HDDS-1696.
RocksDB use separate Write-ahead-log location for OM RocksDB.
URL: https://github.com/apache/hadoop/pull/981#discussion_r294543251
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStoreBuilder.java
##
@@ -220,4 +227,15 @@ private File getDBFile() throws IOException {
return Paths.get(dbPath.toString(), dbname).toFile();
}
+ private File getWALFile() throws IOException {
+if (walPath == null) {
+ LOG.error("Write-ahead-log path is " +
Review comment:
I would think that most people will never even add WAL as an option, so this
should not be an error IMHO.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 261877)
> RocksDB use separate Write-ahead-log location for RocksDB.
> --
>
> Key: HDDS-1696
> URL: https://issues.apache.org/jira/browse/HDDS-1696
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> This will help on production systems where WAL logs and db actual data files
> will be in a different location. During compaction, it will not affect actual
> writes.
>
> Suggested by [~msingh]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1696) RocksDB use separate Write-ahead-log location for RocksDB.
[ https://issues.apache.org/jira/browse/HDDS-1696?focusedWorklogId=261874=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261874 ] ASF GitHub Bot logged work on HDDS-1696: Author: ASF GitHub Bot Created on: 17/Jun/19 22:30 Start Date: 17/Jun/19 22:30 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #981: HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB. URL: https://github.com/apache/hadoop/pull/981#discussion_r294540636 ## File path: hadoop-hdds/common/src/main/resources/ozone-default.xml ## @@ -699,6 +710,17 @@ production environments. + +ozone.scm.db.wal.dir Review comment: same comment as above. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 261874) Time Spent: 1h 10m (was: 1h) > RocksDB use separate Write-ahead-log location for RocksDB. > -- > > Key: HDDS-1696 > URL: https://issues.apache.org/jira/browse/HDDS-1696 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > This will help on production systems where WAL logs and db actual data files > will be in a different location. During compaction, it will not affect actual > writes. > > Suggested by [~msingh] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1696) RocksDB use separate Write-ahead-log location for RocksDB.
[ https://issues.apache.org/jira/browse/HDDS-1696?focusedWorklogId=261875=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261875 ] ASF GitHub Bot logged work on HDDS-1696: Author: ASF GitHub Bot Created on: 17/Jun/19 22:30 Start Date: 17/Jun/19 22:30 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #981: HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB. URL: https://github.com/apache/hadoop/pull/981#discussion_r294543079 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java ## @@ -35,6 +35,7 @@ // metadata dir but in future we may support multiple for redundancy or // performance. public static final String OZONE_SCM_DB_DIRS = "ozone.scm.db.dirs"; + public static final String OZONE_SCM_DB_WAL_DIR = "ozone.scm.db.wal.dir"; Review comment: Do we have data that indicates the compaction is the root cause of issues? On a normal SSD, we have around 80K IOPS are we saying that we are need more bandwidth than that ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 261875) Time Spent: 1h 20m (was: 1h 10m) > RocksDB use separate Write-ahead-log location for RocksDB. > -- > > Key: HDDS-1696 > URL: https://issues.apache.org/jira/browse/HDDS-1696 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > This will help on production systems where WAL logs and db actual data files > will be in a different location. During compaction, it will not affect actual > writes. > > Suggested by [~msingh] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1696) RocksDB use separate Write-ahead-log location for RocksDB.
[ https://issues.apache.org/jira/browse/HDDS-1696?focusedWorklogId=261830=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261830 ] ASF GitHub Bot logged work on HDDS-1696: Author: ASF GitHub Bot Created on: 17/Jun/19 21:32 Start Date: 17/Jun/19 21:32 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #981: HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB. URL: https://github.com/apache/hadoop/pull/981#issuecomment-502859264 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1002 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 90 | Maven dependency ordering for branch | | +1 | mvninstall | 609 | trunk passed | | +1 | compile | 291 | trunk passed | | +1 | checkstyle | 79 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 850 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 164 | trunk passed | | 0 | spotbugs | 360 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 561 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 31 | Maven dependency ordering for patch | | +1 | mvninstall | 507 | the patch passed | | +1 | compile | 304 | the patch passed | | +1 | javac | 304 | the patch passed | | -0 | checkstyle | 38 | hadoop-ozone: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 665 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 163 | the patch passed | | +1 | findbugs | 587 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 109 | hadoop-hdds in the patch failed. | | -1 | unit | 2171 | hadoop-ozone in the patch failed. | | +1 | asflicense | 46 | The patch does not generate ASF License warnings. | | | | 8482 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.utils.db.TestDBStoreBuilder | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestContainerStateMachine | | | hadoop.hdds.scm.safemode.TestSCMSafeModeWithPipelineRules | | | hadoop.ozone.ozShell.TestOzoneShell | | | hadoop.ozone.TestSecureOzoneCluster | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-981/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/981 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux e53471f1b77a 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 3d020e9 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-981/1/artifact/out/diff-checkstyle-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-981/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-981/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-981/1/testReport/ | | Max. process+thread count | 4043 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/server-scm hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-981/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 261830) Time Spent: 20m
