[kudu-CR] util: use uint64 t values for resource limits and adjust usages

2018-03-29 Thread Todd Lipcon (Code Review) 
Todd Lipcon has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9806 )

Change subject: util: use uint64_t values for resource limits and adjust usages
..


Patch Set 3: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7e9bb71af886f732f36a8c93876f47e7591aeadb
Gerrit-Change-Number: 9806
Gerrit-PatchSet: 3
Gerrit-Owner: Adar Dembo 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Fri, 30 Mar 2018 01:29:33 +
Gerrit-HasComments: No

[kudu-CR] util: use uint64 t values for resource limits and adjust usages

2018-03-29 Thread Todd Lipcon (Code Review) 
Todd Lipcon has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9806 )

Change subject: util: use uint64_t values for resource limits and adjust usages
..

util: use uint64_t values for resource limits and adjust usages

Turns out rlim_t is an unsigned 64-bit type, and although a root shell on my
laptop won't let me set RLIMIT_NOFILE particularly high, I can set
RLIMIT_NPROC to any value, and even to RLIM_INFINITY (-1).

So to handle every value correctly, we should really be treating these as
uint64_t rather than int64_t. That also gives us correct RLIM_INFINITY
handling for free, because -1 becomes kuint64max. This also effectively
reverts commit 17aceda.

However, not all callers expect uint64_t values, so some conversion is still
necessary. I also took the opportunity to beef up how the resource limits
were used by incorporating the values of /proc/sys/kernel/pid_max,
/proc/sys/kernel/threads-max, and /proc/sys/fs/file-max as upper bounds.

Change-Id: I7e9bb71af886f732f36a8c93876f47e7591aeadb
Reviewed-on: http://gerrit.cloudera.org:8080/9806
Tested-by: Kudu Jenkins
Reviewed-by: Todd Lipcon 
---
M src/kudu/fs/block_manager.cc
M src/kudu/kserver/kserver.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
4 files changed, 58 insertions(+), 29 deletions(-)

Approvals:
  Kudu Jenkins: Verified
  Todd Lipcon: Looks good to me, approved

--
To view, visit http://gerrit.cloudera.org:8080/9806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I7e9bb71af886f732f36a8c93876f47e7591aeadb
Gerrit-Change-Number: 9806
Gerrit-PatchSet: 4
Gerrit-Owner: Adar Dembo 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon

[kudu-CR] util: use uint64 t values for resource limits and adjust usages

2018-03-28 Thread Adar Dembo (Code Review) 
Hello Kudu Jenkins, Grant Henke, Todd Lipcon,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9806

to look at the new patch set (#3).

Change subject: util: use uint64_t values for resource limits and adjust usages
..

util: use uint64_t values for resource limits and adjust usages

Turns out rlim_t is an unsigned 64-bit type, and although a root shell on my
laptop won't let me set RLIMIT_NOFILE particularly high, I can set
RLIMIT_NPROC to any value, and even to RLIM_INFINITY (-1).

So to handle every value correctly, we should really be treating these as
uint64_t rather than int64_t. That also gives us correct RLIM_INFINITY
handling for free, because -1 becomes kuint64max. This also effectively
reverts commit 17aceda.

However, not all callers expect uint64_t values, so some conversion is still
necessary. I also took the opportunity to beef up how the resource limits
were used by incorporating the values of /proc/sys/kernel/pid_max,
/proc/sys/kernel/threads-max, and /proc/sys/fs/file-max as upper bounds.

Change-Id: I7e9bb71af886f732f36a8c93876f47e7591aeadb
---
M src/kudu/fs/block_manager.cc
M src/kudu/kserver/kserver.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
4 files changed, 58 insertions(+), 29 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/06/9806/3
--
To view, visit http://gerrit.cloudera.org:8080/9806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7e9bb71af886f732f36a8c93876f47e7591aeadb
Gerrit-Change-Number: 9806
Gerrit-PatchSet: 3
Gerrit-Owner: Adar Dembo 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon

[kudu-CR] util: use uint64 t values for resource limits and adjust usages

2018-03-28 Thread Adar Dembo (Code Review) 
Adar Dembo has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9806 )

Change subject: util: use uint64_t values for resource limits and adjust usages
..


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/9806/2/src/kudu/fs/block_manager.cc
File src/kudu/fs/block_manager.cc:

http://gerrit.cloudera.org:8080/#/c/9806/2/src/kudu/fs/block_manager.cc@94
PS2, Line 94: if (ReadFileToString(env, "/proc/sys/fs/file-max", ).ok() 
&&
> what do you think about incorporating this into Env::GetResourceLimit so it
I'd prefer to have the Env call be lightweight and for a wrapper like the kind 
you're describing to sit on top, perhaps in env_util. I think if we end up with 
more usage of this pattern it'd make sense to refactor it into env_util, but 
for now it doesn't matter.


http://gerrit.cloudera.org:8080/#/c/9806/2/src/kudu/util/env_posix.cc
File src/kudu/util/env_posix.cc:

http://gerrit.cloudera.org:8080/#/c/9806/2/src/kudu/util/env_posix.cc@1590
PS2, Line 1590: 2^63
> 2^64 right?
Whoops, was thinking of kint64max when I wrote that.



--
To view, visit http://gerrit.cloudera.org:8080/9806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7e9bb71af886f732f36a8c93876f47e7591aeadb
Gerrit-Change-Number: 9806
Gerrit-PatchSet: 2
Gerrit-Owner: Adar Dembo 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Wed, 28 Mar 2018 21:43:52 +
Gerrit-HasComments: Yes

[kudu-CR] util: use uint64 t values for resource limits and adjust usages

2018-03-28 Thread Todd Lipcon (Code Review) 
Todd Lipcon has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9806 )

Change subject: util: use uint64_t values for resource limits and adjust usages
..


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/9806/2/src/kudu/fs/block_manager.cc
File src/kudu/fs/block_manager.cc:

http://gerrit.cloudera.org:8080/#/c/9806/2/src/kudu/fs/block_manager.cc@94
PS2, Line 94: if (ReadFileToString(env, "/proc/sys/fs/file-max", ).ok() 
&&
what do you think about incorporating this into Env::GetResourceLimit so it's 
not just a wrapper on getrlimit but also takes into account /proc? Logically it 
seems like it's part of the environment... but it's up to you if you disagree, 
I can also see the logic that Env should just wrap a system call.


http://gerrit.cloudera.org:8080/#/c/9806/2/src/kudu/util/env_posix.cc
File src/kudu/util/env_posix.cc:

http://gerrit.cloudera.org:8080/#/c/9806/2/src/kudu/util/env_posix.cc@1590
PS2, Line 1590: 2^63
2^64 right?



--
To view, visit http://gerrit.cloudera.org:8080/9806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7e9bb71af886f732f36a8c93876f47e7591aeadb
Gerrit-Change-Number: 9806
Gerrit-PatchSet: 2
Gerrit-Owner: Adar Dembo 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Wed, 28 Mar 2018 21:40:33 +
Gerrit-HasComments: Yes

[kudu-CR] util: use uint64 t values for resource limits and adjust usages

2018-03-27 Thread Adar Dembo (Code Review) 
Adar Dembo has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9806 )

Change subject: util: use uint64_t values for resource limits and adjust usages
..


Patch Set 1:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/9806/1/src/kudu/kserver/kserver.cc
File src/kudu/kserver/kserver.cc:

http://gerrit.cloudera.org:8080/#/c/9806/1/src/kudu/kserver/kserver.cc@95
PS1, Line 95:   uint64_t rlimit = 
env->GetResourceLimit(Env::ResourceLimitType::RUNNING_THREADS_PER_EUID);
> Not exactly associated with this patch, but should we check /proc/sys/kerne
Done.

I also incorporated /proc/sys/fs/file-max into the block_manager.cc equivalent.


http://gerrit.cloudera.org:8080/#/c/9806/1/src/kudu/util/env_posix.cc
File src/kudu/util/env_posix.cc:

http://gerrit.cloudera.org:8080/#/c/9806/1/src/kudu/util/env_posix.cc@1591
PS1, Line 1591: return l.rlim_cur;
> can we static_assert(RLIM_INFINITY == kuint64Max) or somesuch here? I think
Done


http://gerrit.cloudera.org:8080/#/c/9806/1/src/kudu/util/threadpool.cc
File src/kudu/util/threadpool.cc:

http://gerrit.cloudera.org:8080/#/c/9806/1/src/kudu/util/threadpool.cc@474
PS1, Line 474:   uint64_t capacity_remaining = max_threads_ - active_threads_ +
> it also makes this calculation more error-prone with regard to overflows/un
Yeah, UBSAN flagged this as well.

I've reverted the threadpool changes



--
To view, visit http://gerrit.cloudera.org:8080/9806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7e9bb71af886f732f36a8c93876f47e7591aeadb
Gerrit-Change-Number: 9806
Gerrit-PatchSet: 1
Gerrit-Owner: Adar Dembo 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Tue, 27 Mar 2018 23:50:00 +
Gerrit-HasComments: Yes

[kudu-CR] util: use uint64 t values for resource limits and adjust usages

2018-03-27 Thread Adar Dembo (Code Review) 
Hello Kudu Jenkins, Grant Henke, Todd Lipcon,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9806

to look at the new patch set (#2).

Change subject: util: use uint64_t values for resource limits and adjust usages
..

util: use uint64_t values for resource limits and adjust usages

Turns out rlim_t is an unsigned 64-bit type, and although a root shell on my
laptop won't let me set RLIMIT_NOFILE particularly high, I can set
RLIMIT_NPROC to any value, and even to RLIM_INFINITY (-1).

So to handle every value correctly, we should really be treating these as
uint64_t rather than int64_t. That also gives us correct RLIM_INFINITY
handling for free, because -1 becomes kuint64max. This also effectively
reverts commit 17aceda.

However, not all callers expect uint64_t values, so some conversion is still
necessary. I also took the opportunity to beef up how the resource limits
were used by incorporating the values of /proc/sys/kernel/pid_max,
/proc/sys/kernel/threads-max, and /proc/sys/fs/file-max as upper bounds.

Change-Id: I7e9bb71af886f732f36a8c93876f47e7591aeadb
---
M src/kudu/fs/block_manager.cc
M src/kudu/kserver/kserver.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
4 files changed, 58 insertions(+), 29 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/06/9806/2
--
To view, visit http://gerrit.cloudera.org:8080/9806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7e9bb71af886f732f36a8c93876f47e7591aeadb
Gerrit-Change-Number: 9806
Gerrit-PatchSet: 2
Gerrit-Owner: Adar Dembo 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon