[kudu-CR] transaction tracker: back-off when logging in-flight transactions

2016-09-01 Thread Todd Lipcon (Code Review) 
Todd Lipcon has submitted this change and it was merged.

Change subject: transaction_tracker: back-off when logging in-flight 
transactions
..


transaction_tracker: back-off when logging in-flight transactions

On a test cluster pushed into overload, I ended up with thousands of
in-flight transactions which were only being drained at the rate of one
or two per second. Dumping the entire list of transactions once a second
ended up flooding the glog with many MB per second work of transaction
strings.

This changes the dumping to back-off exponentially.

Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
Reviewed-on: http://gerrit.cloudera.org:8080/4159
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin 
Reviewed-by: David Ribeiro Alves 
---
M src/kudu/tablet/transactions/transaction_tracker.cc
1 file changed, 16 insertions(+), 13 deletions(-)

Approvals:
  David Ribeiro Alves: Looks good to me, approved
  Alexey Serbin: Looks good to me, but someone else must approve
  Kudu Jenkins: Verified



-- 
To view, visit http://gerrit.cloudera.org:8080/4159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy 
Gerrit-Reviewer: Todd Lipcon

[kudu-CR] transaction tracker: back-off when logging in-flight transactions

2016-09-01 Thread David Ribeiro Alves (Code Review) 
David Ribeiro Alves has posted comments on this change.

Change subject: transaction_tracker: back-off when logging in-flight 
transactions
..


Patch Set 1: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4159/1/src/kudu/tablet/transactions/transaction_tracker.cc
File src/kudu/tablet/transactions/transaction_tracker.cc:

Line 207:   MonoTime next_log_time = start_time + MonoDelta::FromSeconds(1);
> not sure how to do that in a nice way since we're also using the times for 
fair enough


-- 
To view, visit http://gerrit.cloudera.org:8080/4159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: Yes

[kudu-CR] transaction tracker: back-off when logging in-flight transactions

2016-08-31 Thread Todd Lipcon (Code Review) 
Todd Lipcon has posted comments on this change.

Change subject: transaction_tracker: back-off when logging in-flight 
transactions
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4159/1/src/kudu/tablet/transactions/transaction_tracker.cc
File src/kudu/tablet/transactions/transaction_tracker.cc:

Line 207:   MonoTime next_log_time = start_time + MonoDelta::FromSeconds(1);
> can you pull the log backoff logic into a generic logging helper, seems lik
not sure how to do that in a nice way since we're also using the times for 
timeout calculation, etc


-- 
To view, visit http://gerrit.cloudera.org:8080/4159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: Yes

[kudu-CR] transaction tracker: back-off when logging in-flight transactions

2016-08-31 Thread David Ribeiro Alves (Code Review) 
David Ribeiro Alves has posted comments on this change.

Change subject: transaction_tracker: back-off when logging in-flight 
transactions
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4159/1/src/kudu/tablet/transactions/transaction_tracker.cc
File src/kudu/tablet/transactions/transaction_tracker.cc:

Line 207:   MonoTime next_log_time = start_time + MonoDelta::FromSeconds(1);
can you pull the log backoff logic into a generic logging helper, seems like it 
would be useful elsewhere


-- 
To view, visit http://gerrit.cloudera.org:8080/4159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy 
Gerrit-HasComments: Yes

[kudu-CR] transaction tracker: back-off when logging in-flight transactions

2016-08-29 Thread Alexey Serbin (Code Review) 
Alexey Serbin has posted comments on this change.

Change subject: transaction_tracker: back-off when logging in-flight 
transactions
..


Patch Set 1: Code-Review+1

-- 
To view, visit http://gerrit.cloudera.org:8080/4159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy 
Gerrit-HasComments: No

[kudu-CR] transaction tracker: back-off when logging in-flight transactions

2016-08-29 Thread Kudu Jenkins (Code Review) 
Kudu Jenkins has posted comments on this change.

Change subject: transaction_tracker: back-off when logging in-flight 
transactions
..


Patch Set 1:

Build Started http://104.196.14.100/job/kudu-gerrit/3133/

-- 
To view, visit http://gerrit.cloudera.org:8080/4159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy 
Gerrit-HasComments: No

[kudu-CR] transaction tracker: back-off when logging in-flight transactions

2016-08-29 Thread Todd Lipcon (Code Review) 
Hello Mike Percy,

I'd like you to do a code review.  Please visit

http://gerrit.cloudera.org:8080/4159

to review the following change.

Change subject: transaction_tracker: back-off when logging in-flight 
transactions
..

transaction_tracker: back-off when logging in-flight transactions

On a test cluster pushed into overload, I ended up with thousands of
in-flight transactions which were only being drained at the rate of one
or two per second. Dumping the entire list of transactions once a second
ended up flooding the glog with many MB per second work of transaction
strings.

This changes the dumping to back-off exponentially.

Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
---
M src/kudu/tablet/transactions/transaction_tracker.cc
1 file changed, 16 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/59/4159/1
-- 
To view, visit http://gerrit.cloudera.org:8080/4159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I7d1a2563dd57c59ae8c130a5b26966f4964048de
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: Mike Percy