[kudu-CR] fs: update default data dir group size

2018-01-13 Thread Adar Dembo (Code Review) 
Adar Dembo has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8995 )

Change subject: fs: update default data dir group size
..


Patch Set 5: Code-Review+1


--
To view, visit http://gerrit.cloudera.org:8080/8995
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2dd0d3f8cf140f684318c0dffb45a1091302ecdc
Gerrit-Change-Number: 8995
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Wong 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Andrew Wong 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Sun, 14 Jan 2018 06:16:11 +
Gerrit-HasComments: No

[kudu-CR] KUDU-1489: allow configuration of metadata dir

2018-01-13 Thread Andrew Wong (Code Review) 
Hello Tidy Bot, Kudu Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9027

to look at the new patch set (#3).

Change subject: KUDU-1489: allow configuration of metadata dir
..

KUDU-1489: allow configuration of metadata dir

Metadata files currently reside in the first configured data directory,
which isn't always the best choice; this first drive, if not performant,
can act as a bottleneck. Often times the drive containing the WALs is a
better choice, as it is recommended to be the fastest.

This patch allows users to configure their metadata directory through
the new fs_metadata_dir flag. If empty, Kudu will check if a metadata
directory exists in the first member of fs_data_dirs, and if none
exists, Kudu will use fs_wal_dir as the root directory for metadata.
Otherwise, the flag will be honored verbatim.

It is up to the user to take caution in changing this flag; updating the
flag without also manually moving any existing metadata will cause Kudu
to fail at startup.

Aside from the update to the directory location, codepaths that
previously assumed that the first data directory _must_ be healthy have
been updated. The remaining invariant is that at least a single data
directory must be healthy.

A new test cases is added to fs_manager-test, and a test case in
data_dirs-test is updated as a sanity check to show that we can now open
the directory manager with a failed first data directory (previously
this codepath would hit D/CHECK failures).

Change-Id: I375c6b2eb283db5fa9c956135d98252c8781f5db
---
M src/kudu/fs/block_manager_util.cc
M src/kudu/fs/data_dirs-test.cc
M src/kudu/fs/data_dirs.cc
M src/kudu/fs/fs_manager-test.cc
M src/kudu/fs/fs_manager.cc
M src/kudu/fs/fs_manager.h
6 files changed, 143 insertions(+), 40 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/27/9027/3
--
To view, visit http://gerrit.cloudera.org:8080/9027
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I375c6b2eb283db5fa9c956135d98252c8781f5db
Gerrit-Change-Number: 9027
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Wong 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] fs: update default data dir group size

2018-01-13 Thread Andrew Wong (Code Review) 
Andrew Wong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8995 )

Change subject: fs: update default data dir group size
..


Patch Set 5:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/8995/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/8995/3//COMMIT_MSG@10
PS3, Line 10: sets the new default to 3. Upon experimenting with the flag via 
YCSB
> I think we should also try an analytic workload like tpch_real_world to loa
Ack


http://gerrit.cloudera.org:8080/#/c/8995/3/src/kudu/fs/data_dirs.cc
File src/kudu/fs/data_dirs.cc:

http://gerrit.cloudera.org:8080/#/c/8995/3/src/kudu/fs/data_dirs.cc@69
PS3, Line 69:  "data dirs. Using fewer data dirs per tablet means a 
single "
> we should also document the effect this setting has on fault tolerance
Done



--
To view, visit http://gerrit.cloudera.org:8080/8995
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2dd0d3f8cf140f684318c0dffb45a1091302ecdc
Gerrit-Change-Number: 8995
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Wong 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Andrew Wong 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Sat, 13 Jan 2018 17:43:43 +
Gerrit-HasComments: Yes

[kudu-CR] fs: update default data dir group size

2018-01-13 Thread Andrew Wong (Code Review) 
Hello Kudu Jenkins, Adar Dembo, Todd Lipcon,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/8995

to look at the new patch set (#5).

Change subject: fs: update default data dir group size
..

fs: update default data dir group size

This patch makes fs_target_data_dirs_per_tablet non-experimental and
sets the new default to 3. Upon experimenting with the flag via YCSB
workloads, only read workload tail latency seemed to be affected at very
small group sizes (e.g. 1). 3 seems like a reasonable choice, and we can
always update it in the future.

Note that existing data will not be updated with the new flag. Only new
tablets will honor the sizing.

Change-Id: I2dd0d3f8cf140f684318c0dffb45a1091302ecdc
---
M src/kudu/fs/data_dirs.cc
M src/kudu/tserver/tablet_server-test.cc
2 files changed, 13 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/95/8995/5
--
To view, visit http://gerrit.cloudera.org:8080/8995
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2dd0d3f8cf140f684318c0dffb45a1091302ecdc
Gerrit-Change-Number: 8995
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Wong 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Andrew Wong 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon

[kudu-CR] KUDU-1489: allow configuration of metadata dir

2018-01-13 Thread Andrew Wong (Code Review) 
Andrew Wong has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9027


Change subject: KUDU-1489: allow configuration of metadata dir
..

KUDU-1489: allow configuration of metadata dir

Metadata files currently reside in the first configured data directory,
which isn't always the best choice. This first drive, if not performant,
can act as a bottleneck; often times the drive containing the WALs is a
better choice, as it is recommended to be the fastest.

This patch allows users to configure their metadata directory through
the new fs_metadata_dir flag. If empty, Kudu will check if a metadata
directory exists in the first member of fs_data_dirs, and if none
exists, Kudu will use fs_wal_dir as the root directory for metadata.
Otherwise, the flag will be honored verbatim.

It is up to the user to take caution in changing this flag; updating the
flag without also manually moving any existing metadata will cause Kudu
to fail at startup.

Aside from the update to the directory location, codepaths that
previously assumed that the first data directory _must_ be healthy have
been updated. The remaining invariant is that at least a single data
directory must be healthy.

A new test cases is added to fs_manager-test, and a test case in
data_dirs-test is updated as a sanity check to show that we can now open
the directory manager with a failed first data directory (previously
this codepath with lead to D/CHECK failures).

Change-Id: I375c6b2eb283db5fa9c956135d98252c8781f5db
---
M src/kudu/fs/block_manager_util.cc
M src/kudu/fs/data_dirs-test.cc
M src/kudu/fs/data_dirs.cc
M src/kudu/fs/fs_manager-test.cc
M src/kudu/fs/fs_manager.cc
M src/kudu/fs/fs_manager.h
6 files changed, 142 insertions(+), 40 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/27/9027/1
--
To view, visit http://gerrit.cloudera.org:8080/9027
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I375c6b2eb283db5fa9c956135d98252c8781f5db
Gerrit-Change-Number: 9027
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong

[kudu-CR] fs: update default data dir group size

2018-01-13 Thread Andrew Wong (Code Review) 
Hello Kudu Jenkins, Adar Dembo, Todd Lipcon,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/8995

to look at the new patch set (#4).

Change subject: fs: update default data dir group size
..

fs: update default data dir group size

This patch makes fs_target_data_dirs_per_tablet non-experimental and
sets the new default to 3. Upon experimenting with the flag via YCSB
workloads, only read workload tail latency seemed to be affected at very
small group sizes (e.g. 1). 3 seems like a reasonable choice, and we can
always update it in the future.

Note that existing data will not be updated with the new flag. Only new
tablets will honor the sizing.

Change-Id: I2dd0d3f8cf140f684318c0dffb45a1091302ecdc
---
M src/kudu/fs/data_dirs.cc
M src/kudu/tserver/tablet_server-test.cc
2 files changed, 8 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/95/8995/4
--
To view, visit http://gerrit.cloudera.org:8080/8995
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2dd0d3f8cf140f684318c0dffb45a1091302ecdc
Gerrit-Change-Number: 8995
Gerrit-PatchSet: 4
Gerrit-Owner: Andrew Wong 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Andrew Wong 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon