[jira] [Commented] (HDDS-359) RocksDB Profiles support
[ https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595402#comment-16595402 ] Hudson commented on HDDS-359: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14848 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14848/]) HDDS-359. RocksDB Profiles support. Contributed by Anu Engineer. (elek: rev c61824a18940ef37dc7201717a3115a78bf942d4) * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java * (edit) hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestRDBStore.java * (add) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBProfile.java * (add) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBConfigFromFile.java * (add) hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestDBStoreBuilder.java * (edit) hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestRDBTableStore.java * (add) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TableConfig.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStore.java * (add) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStoreBuilder.java * (add) hadoop-hdds/common/src/test/resources/test.db.ini * (edit) hadoop-hdds/common/src/main/resources/ozone-default.xml * (add) hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestDBConfigFromFile.java * (edit) hadoop-hdds/pom.xml > RocksDB Profiles support > > > Key: HDDS-359 > URL: https://issues.apache.org/jira/browse/HDDS-359 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Anu Engineer >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-359.001.patch, HDDS-359.002.patch, > HDDS-359.003.patch, HDDS-359.004.patch, HDDS-359.005.patch > > > This allows us to tune the OM/SCM DB for different machine configurations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[ https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595375#comment-16595375 ] Anu Engineer commented on HDDS-359: --- [~elek] Thanks for the commit. [~xyao], [~elek] Thanks for the comments. > RocksDB Profiles support > > > Key: HDDS-359 > URL: https://issues.apache.org/jira/browse/HDDS-359 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Anu Engineer >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-359.001.patch, HDDS-359.002.patch, > HDDS-359.003.patch, HDDS-359.004.patch, HDDS-359.005.patch > > > This allows us to tune the OM/SCM DB for different machine configurations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[ https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595307#comment-16595307 ] Elek, Marton commented on HDDS-359: --- +1 Will commit it to the trunk shortly, if no more objections. > RocksDB Profiles support > > > Key: HDDS-359 > URL: https://issues.apache.org/jira/browse/HDDS-359 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Anu Engineer >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-359.001.patch, HDDS-359.002.patch, > HDDS-359.003.patch, HDDS-359.004.patch, HDDS-359.005.patch > > > This allows us to tune the OM/SCM DB for different machine configurations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[
https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594145#comment-16594145
]
genericqa commented on HDDS-359:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdds {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdds {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
18s{color} | {color:green} hadoop-hdds in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
8s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDDS-359 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12937320/HDDS-359.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 7a927d46e7cc 4.4.0-133-generic #159-Ubuntu
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[
https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594066#comment-16594066
]
Anu Engineer commented on HDDS-359:
---
[~elek] Thanks for the comments. Patch v5 addresses all comments.
{quote}Would be great to get some log output about initializing the
table/columnfamiliy with a specific file/profile
{quote}
Done DbStoreBuilder will write all that information to Log at info level now.
{quote} I would use HddsConfigKeys instead of OzoneConfigKeys. The
functionality is part of the core hdds and long term we need to separated the
ozone specific and hdds specific configuration
{quote}
Very good catch. Thanks for noticing that issue. I have moved the key to a
better location. Please take a look at patch v5 when you get a chance.
> RocksDB Profiles support
>
>
> Key: HDDS-359
> URL: https://issues.apache.org/jira/browse/HDDS-359
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Anu Engineer
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-359.001.patch, HDDS-359.002.patch,
> HDDS-359.003.patch, HDDS-359.004.patch, HDDS-359.005.patch
>
>
> This allows us to tune the OM/SCM DB for different machine configurations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[
https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593392#comment-16593392
]
Elek, Marton commented on HDDS-359:
---
Thanks Anu the patch. I like this approach especially the flexibility in the
configuration method. As you wrote:
{quote}
Here is what we are doing, if there is a name.db.ini in the config directory,
we will use those params for the RocksDB. Otherwise, we will use the profile
key if specified, if there is nothing specified we will use the default
profile. The return of null makes it easy for us to know that we need to get a
profile from the secondary sources.
{quote}
I have two minor comments (non of them are blockers):
1. Would be great to get some log output about initializing the
table/columnfamiliy with a specific file/profile. Users may not familiar with
this configuration but checking the log it could be a warning if I have only
spinning disk but the log contains something about SSD. Also would be easier to
check if a custom db file is picked. I would be happy even with an INFO level
line.
2. I would use HddsConfigKeys instead of OzoneConfigKeys. The functionality is
part of the core hdds and long term we need to separated the ozone specific and
hdds specific configuration (not a blocker as we have a lot of other
configurations which should be moved to the HddsConfigKeys).
But non of these are blockers, both of them could be improved later. I am +1
even without these...
> RocksDB Profiles support
>
>
> Key: HDDS-359
> URL: https://issues.apache.org/jira/browse/HDDS-359
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Anu Engineer
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-359.001.patch, HDDS-359.002.patch,
> HDDS-359.003.patch, HDDS-359.004.patch
>
>
> This allows us to tune the OM/SCM DB for different machine configurations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[
https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592256#comment-16592256
]
genericqa commented on HDDS-359:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdds {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdds {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
23s{color} | {color:green} hadoop-hdds in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
22s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDDS-359 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12937065/HDDS-359.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 05677be3a378 4.4.0-133-generic #159-Ubuntu
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[
https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1659#comment-1659
]
Anu Engineer commented on HDDS-359:
---
{quote}I think the confusion is that DBStoreBuilder#addTable() allows add per
table column family option. But the caller processTableNames() only pass the
same one from dbProfile.getColumnFamilyOptions() in line 141 based on SSD/DISK.
{quote}
I see the issue you are mentioning and I can see the source of confusion this
creating. When we call the function addTable(string, option), internally we are
adding that to a set.
{{private Set tables;}}
However, when the user makes a call to addTable(string) where the user does not
specify an Option, we add that a list of Strings.
{{private List tableNames;}}
Later in the "processTables" call, we walk the tableNames and make calls into
addTable(string, default Option). The addTable with name only makes some code
simpler, especially if you are happy with the defaults.
bq. I mean the TableConfig class is almost the same as ColumnFamilyOption. We
could potentially use it directly.
Yes, that would mean that we have to send into 2 lists, one forthe name and
another for ColumnFamilyOption. This is class creates a relationship between
those 2 variables.
> RocksDB Profiles support
>
>
> Key: HDDS-359
> URL: https://issues.apache.org/jira/browse/HDDS-359
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Anu Engineer
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-359.001.patch, HDDS-359.002.patch,
> HDDS-359.003.patch, HDDS-359.004.patch
>
>
> This allows us to tune the OM/SCM DB for different machine configurations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[
https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592214#comment-16592214
]
Xiaoyu Yao commented on HDDS-359:
-
{quote}{quote}Line 141: this assumes all tables have the same column family
option. Is it possible to have per table column family option
{quote}
We already do that. When the user add a table, they can call addTable(string)
or addTable(string, options). If the first addTable is used, we iterate thru
the string table and add the default option. If the user had called
addTable(string, option) then the user specified option would be used.
{quote}
I think the confusion is that DBStoreBuilder#addTable() allows add per table
column family option. But the caller processTableNames() only pass the same one
from dbProfile.getColumnFamilyOptions() in line 141 based on SSD/DISK.
For example, if we have two tables: one for normal key and the other for open
key both take the SSD profile but open key table may want some different column
family settings.
{quote}{quote}Can we abstract the table config otherwise we can use
ColumnFamilyOptions directly
{quote}
Sorry, I am not sure I understand this comment clearly. Could you please
explain it to me once more ?
{quote}
I mean the TableConfig class is almost the same as ColumnFamilyOption. We could
potentially use it directly.
{quote}It is hard to merge these since it because harder to define what the
final config will look like.
{quote}
Good point. Let's revisit this later if things change.
> RocksDB Profiles support
>
>
> Key: HDDS-359
> URL: https://issues.apache.org/jira/browse/HDDS-359
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Anu Engineer
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-359.001.patch, HDDS-359.002.patch,
> HDDS-359.003.patch, HDDS-359.004.patch
>
>
> This allows us to tune the OM/SCM DB for different machine configurations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[
https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592163#comment-16592163
]
Anu Engineer commented on HDDS-359:
---
[~xyao] Thanks for the review. The next patch addresses some of the comments.
{quote}DBConfigFromFile.java Line 122: can we move this to line 117 so that the
caller does not need to check null.
{quote}
Here is what we are doing, if there is a name.db.ini in the config directory,
we will use those params for the RocksDB. Otherwise we will use the profile key
if specified, if there is nothing specified we will use the default profile.
The return of null makes it easy for us to know that we need to get profile
from the secondary sources.
{quote}Line 79: NIT: annotate for test only?
{quote}
addTable is not test only, if you add a table without columnfamily, we will
default to profiles.
{quote}Line 118: NIT: rename from processTableNames => processTables?
{quote}
Done.
{quote}Line 141: this assumes all tables have the same column family option. Is
it possible to have per table column family option
{quote}
We already do that. When the user add a table, they can call addTable(string)
or addTable(string, options). If the first addTable is used, we iterate thru
the string table and add the default option. If the user had called
addTable(string, option) then the user specified option would be used.
{quote}Line 144: should we document in the build class that addTable() should
not add
{quote}
I can do that, but if the user tried to add an existing table we will throw an
exception. we have a test case for that case.
{quote}Line 148-170: the current code loads from rocksDBOption parameter, OR
config file OR pre-defined db profiles? Is it possible to allow all of these
and merge/overwrite with the config file like hadoop configs.
{quote}
The current code attempts to get config in a heireachical fashion.
First it looks for the config from an .INI file, if not found, it will try to
read the profile from ozone-deafult.xml and if that is not defined, we will use
the default profile. We don't try to merge the configs, but use each one. It is
hard to merge these since it because harder to define what the final config
will look like.
{quote}Can we abstract the table config otherwise we can use
ColumnFamilyOptions directly
{quote}
Sorry, I am not sure I understand this comment clearly. Could you please
explain it to me once more ?
{quote}Test.db.ini : Add an exception to the license check plugin?
{quote}
done.
{quote}Line 67: should we clean up the copied test ini file in the tearDown?
{quote}
we are copying a file into the Temporary folder. Junit will clean it up
automatically at the exit of the test.
> RocksDB Profiles support
>
>
> Key: HDDS-359
> URL: https://issues.apache.org/jira/browse/HDDS-359
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Anu Engineer
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-359.001.patch, HDDS-359.002.patch,
> HDDS-359.003.patch, HDDS-359.004.patch
>
>
> This allows us to tune the OM/SCM DB for different machine configurations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[
https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591859#comment-16591859
]
Xiaoyu Yao commented on HDDS-359:
-
Thanks [~anu] for working on this. The patch looks good to me overall. Just a
few minor comments:
DBConfigFromFile.java
Line 122: can we move this to line 117 so that the caller does not need to
check null. I see there are logic later in DBStoreBuild.java relies on null
check. This can be addressed later with the comments for Line 148-170 in
DBStoreBuilder.java.
DBStoreBuilder.java
Line 79: NIT: annotate for test only?
Line 118: NIT: rename from processTableNames => processTables?
Line 141: this assumes all tables have the same column family option. Is it
possible to have per table column family option (either fully customized or
predefined set of column family options)
{color:#00}Line 144: should we document in the build class that addTable()
should not add "{color}{color:#658aba}default" column family explicitly?{color}
Line 148-170: the current code loads from rocksDBOption parameter, OR config
file OR pre-defined db profiles? Is it possible to allow all of these and
merge/overwrite with the config file like hadoop configs. (This can be added
later with a separate JIRA)
TableConfig.java
Can we abstract the table config otherwise we can use ColumnFamilyOptions
directly?
Test.db.ini
Add an exception to the license check plugin?
TestDBConfigFromFile.java
Line 67: should we clean up the copied test ini file in the tearDown?
> RocksDB Profiles support
>
>
> Key: HDDS-359
> URL: https://issues.apache.org/jira/browse/HDDS-359
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Anu Engineer
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-359.001.patch, HDDS-359.002.patch,
> HDDS-359.003.patch
>
>
> This allows us to tune the OM/SCM DB for different machine configurations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[ https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591312#comment-16591312 ] Elek, Marton commented on HDDS-359: --- [~anu] Yetus tries to minimalize the number of unit tests. It checks the changed maven projects and tries to calculate the required tests. If ozone-integration project is not changed, the tests won't be executed. To modify this behaviour we need to adjust the hadoop yetus presonality (which is stored in the yetus repository). > RocksDB Profiles support > > > Key: HDDS-359 > URL: https://issues.apache.org/jira/browse/HDDS-359 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Anu Engineer >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-359.001.patch, HDDS-359.002.patch, > HDDS-359.003.patch > > > This allows us to tune the OM/SCM DB for different machine configurations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[
https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591025#comment-16591025
]
genericqa commented on HDDS-359:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
51s{color} | {color:green} common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
21s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDDS-359 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12936925/HDDS-359.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 2c9707e623ca 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ca29fb7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/825/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HDDS-Build/825/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 412 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/common U: hadoop-hdds/common |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/825/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[ https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590978#comment-16590978 ] Anu Engineer commented on HDDS-359: --- I am posting the next version of this patch for the Jenkins did not run the tests correctly. I have noticed a failure and for some reason Jenkins did not run all the Ozone tests. cc: [~elek], [~xyao], [~msingh], [~nandakumar131] > RocksDB Profiles support > > > Key: HDDS-359 > URL: https://issues.apache.org/jira/browse/HDDS-359 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Anu Engineer >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-359.001.patch, HDDS-359.002.patch > > > This allows us to tune the OM/SCM DB for different machine configurations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[ https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590977#comment-16590977 ] Anu Engineer commented on HDDS-359: --- Asf license warning is for a file which I have copied from RocksDB examples. Even though RocksDB is Apache licensed, this file does not have a Apache header in the RocksDB repo. So I am not adding a Apache header since I did not author the file, only made some edits. I will add an RAT file exception later in another patch. > RocksDB Profiles support > > > Key: HDDS-359 > URL: https://issues.apache.org/jira/browse/HDDS-359 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Anu Engineer >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-359.001.patch, HDDS-359.002.patch > > > This allows us to tune the OM/SCM DB for different machine configurations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[
https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590810#comment-16590810
]
genericqa commented on HDDS-359:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 19m
44s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
19s{color} | {color:green} common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
26s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDDS-359 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12936878/HDDS-359.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 96ce74c38cf4 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ca29fb7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/820/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HDDS-Build/820/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 443 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/common U: hadoop-hdds/common |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/820/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> RocksDB Profiles support
>
>
> Key: HDDS-359
> URL:
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[ https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590737#comment-16590737 ] Anu Engineer commented on HDDS-359: --- [~xyao] Done, I have rebased the patch to trunk. > RocksDB Profiles support > > > Key: HDDS-359 > URL: https://issues.apache.org/jira/browse/HDDS-359 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Anu Engineer >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-359.001.patch, HDDS-359.002.patch > > > This allows us to tune the OM/SCM DB for different machine configurations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[ https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589686#comment-16589686 ] Xiaoyu Yao commented on HDDS-359: - [~anu], can you rebase the patch as it does not apply any more. Thanks! > RocksDB Profiles support > > > Key: HDDS-359 > URL: https://issues.apache.org/jira/browse/HDDS-359 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Anu Engineer >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-359.001.patch > > > This allows us to tune the OM/SCM DB for different machine configurations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[ https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588334#comment-16588334 ] Anu Engineer commented on HDDS-359: --- This patch depends on HDDS-356. Hence not submitting now. > RocksDB Profiles support > > > Key: HDDS-359 > URL: https://issues.apache.org/jira/browse/HDDS-359 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Anu Engineer >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-359.001.patch > > > This allows us to tune the OM/SCM DB for different machine configurations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
