The branch main has been updated by rrs: URL: https://cgit.FreeBSD.org/src/commit/?id=a9696510f5caef4a7c525fcf4359597798829350
commit a9696510f5caef4a7c525fcf4359597798829350 Author: Randall Stewart <[email protected]> AuthorDate: 2022-02-07 11:37:46 +0000 Commit: Randall Stewart <[email protected]> CommitDate: 2022-02-07 11:37:46 +0000 tcp: Add hystart++ to our cubic implementation. As promised to the transport call on 11/4/22 here is an implementation of hystart++ for cubic. It also cleans up the tcp_congestion function to have a better name. Common variables are moved into the general cc.h structure so that both cubic and newreno can use them for hystart++ Reviewed by: Michael Tuexen, Richard Scheffenegger Sponsored by: Netflix Inc. Differential Revision: https://reviews.freebsd.org/D33035 --- share/man/man4/cc_newreno.4 | 39 +---- share/man/man4/mod_cc.4 | 25 +++- sys/netinet/cc/cc.c | 41 ++++++ sys/netinet/cc/cc.h | 12 ++ sys/netinet/cc/cc_cubic.c | 291 +++++++++++++++++++++++++++++++++----- sys/netinet/cc/cc_cubic.h | 49 +++++++ sys/netinet/cc/cc_newreno.c | 122 +++++----------- sys/netinet/cc/cc_newreno.h | 4 - sys/netinet/tcp_stacks/rack.c | 105 +++++++++----- sys/netinet/tcp_stacks/tcp_rack.h | 8 ++ 10 files changed, 492 insertions(+), 204 deletions(-) diff --git a/share/man/man4/cc_newreno.4 b/share/man/man4/cc_newreno.4 index 1c4f4a39cb36..71d3a239ecd1 100644 --- a/share/man/man4/cc_newreno.4 +++ b/share/man/man4/cc_newreno.4 @@ -75,21 +75,10 @@ the congestion window in response to an ECN congestion signal when .Va net.inet.tcp.cc.abe=1 per: cwnd = (cwnd * CC_NEWRENO_BETA_ECN) / 100. Default is 80. -.It Va CC_NEWRENO_ENABLE_HYSTART -will enable or disable the application of Hystart++. -The current implementation allows the values 0, 1, 2 and 3. -A value of 0 (the default) disables the use of Hystart++. -Setting the value to 1 enables Hystart++. -Setting the value to 2 enables Hystart++ but also will cause, on exit from Hystart++'s CSS, to -set the cwnd to the value of where the increase in RTT first began as -well as setting ssthresh to the flight at send when we exit CSS. -Setting a value of 3 will keep the setting of the cwnd the same as 2, but will cause ssthresh -to be set to the average value between the lowest fas rtt (the value cwnd is -set to) and the fas value at exit of CSS. .PP Note that currently the only way to enable hystart++ is to enable it via socket option. -When enabling it a value of 1 will enable precise internet-draft behavior +When enabling it a value of 1 will enable precise internet-draft (version 4) behavior (subject to any MIB variable settings), other setting (2 and 3) are experimental. .El .PP @@ -120,32 +109,6 @@ the congestion window in response to an ECN congestion signal when .Va net.inet.tcp.cc.abe=1 per: cwnd = (cwnd * beta_ecn) / 100. Default is 80. -.It Va hystartplusplus.bblogs -This boolean controls if black box logging will be done for hystart++ events. If set -to zero (the default) no logging is performed. -If set to one then black box logs will be generated on all hystart++ events. -.It Va hystartplusplus.css_rounds -This value controls the number of rounds that CSS runs for. -The default value matches the current internet-draft of 5. -.It Va hystartplusplus.css_growth_div -This value controls the divisor applied to slowstart during CSS. -The default value matches the current internet-draft of 4. -.It Va hystartplusplus.n_rttsamples -This value controls how many rtt samples must be collected in each round for -hystart++ to be active. -The default value matches the current internet-draft of 8. -.It Va hystartplusplus.maxrtt_thresh -This value controls the maximum rtt variance clamp when considering if CSS is needed. -The default value matches the current internet-draft of 16000 (in microseconds). -For further explanation please see the internet-draft. -.It Va hystartplusplus.minrtt_thresh -This value controls the minimum rtt variance clamp when considering if CSS is needed. -The default value matches the current internet-draft of 4000 (in microseconds). -For further explanation please see the internet-draft. -.It Va hystartplusplus.lowcwnd -This value controls what is the lowest congestion window that the tcp -stack must be at before hystart++ engages. -The default value matches the current internet-draft of 16. .El .Sh SEE ALSO .Xr cc_cdg 4 , diff --git a/share/man/man4/mod_cc.4 b/share/man/man4/mod_cc.4 index 84edf0b2a8ed..ddc7440a4ca7 100644 --- a/share/man/man4/mod_cc.4 +++ b/share/man/man4/mod_cc.4 @@ -83,7 +83,7 @@ The framework exposes the following variables in the branch of the .Xr sysctl 3 MIB: -.Bl -tag -width ".Va abe_frlossreduce" +.Bl -tag -width ".Va hystartplusplus.css_growth_div" .It Va available Read-only list of currently available congestion control algorithms by name. .It Va algorithm @@ -102,10 +102,33 @@ support for ABE and for configuration details. .It Va abe_frlossreduce If non-zero, apply standard beta instead of ABE-beta during ECN-signalled congestion recovery episodes if loss also needs to be repaired. +.It Va hystartplusplus.bblogs +This boolean controls if black box logging will be done for hystart++ events. +If set to zero (the default) no logging is performed. +If set to one then black box logs will be generated on all hystart++ events. +.It Va hystartplusplus.css_rounds +This value controls the number of rounds that CSS runs for. +The default value matches the current internet-draft of 5. +.It Va hystartplusplus.css_growth_div +This value controls the divisor applied to slowstart during CSS. +The default value matches the current internet-draft of 4. +.It Va hystartplusplus.n_rttsamples +This value controls how many rtt samples must be collected in each round for +hystart++ to be active. +The default value matches the current internet-draft of 8. +.It Va hystartplusplus.maxrtt_thresh +This value controls the maximum rtt variance clamp when considering if CSS is needed. +The default value matches the current internet-draft of 16000 (in microseconds). +For further explanation please see the internet-draft. +.It Va hystartplusplus.minrtt_thresh +This value controls the minimum rtt variance clamp when considering if CSS is needed. +The default value matches the current internet-draft of 4000 (in microseconds). +For further explanation please see the internet-draft. .El .Pp Each congestion control module may also expose other MIB variables to control their behaviour. +Note that both newreno and cubic now support hystart++ based on the version 3 of the internet-draft. .Sh Kernel Configuration .Pp All of the available congestion control modules may also be loaded diff --git a/sys/netinet/cc/cc.c b/sys/netinet/cc/cc.c index bcda9276d4d9..647ecb47f18d 100644 --- a/sys/netinet/cc/cc.c +++ b/sys/netinet/cc/cc.c @@ -84,6 +84,13 @@ __FBSDID("$FreeBSD$"); #define CC_DEFAULT "newreno" #endif +uint32_t hystart_minrtt_thresh = 4000; +uint32_t hystart_maxrtt_thresh = 16000; +uint32_t hystart_n_rttsamples = 8; +uint32_t hystart_css_growth_div = 4; +uint32_t hystart_css_rounds = 5; +uint32_t hystart_bblogs = 0; + MALLOC_DEFINE(M_CC_MEM, "CC Mem", "Congestion Control State memory"); /* @@ -582,6 +589,40 @@ SYSCTL_PROC(_net_inet_tcp_cc, OID_AUTO, available, NULL, 0, cc_list_available, "A", "List available congestion control algorithms"); +SYSCTL_NODE(_net_inet_tcp_cc, OID_AUTO, hystartplusplus, + CTLFLAG_RW | CTLFLAG_MPSAFE, NULL, + "New Reno related HyStart++ settings"); + +SYSCTL_UINT(_net_inet_tcp_cc_hystartplusplus, OID_AUTO, minrtt_thresh, + CTLFLAG_RW, + &hystart_minrtt_thresh, 4000, + "HyStarts++ minimum RTT thresh used in clamp (in microseconds)"); + +SYSCTL_UINT(_net_inet_tcp_cc_hystartplusplus, OID_AUTO, maxrtt_thresh, + CTLFLAG_RW, + &hystart_maxrtt_thresh, 16000, + "HyStarts++ maximum RTT thresh used in clamp (in microseconds)"); + +SYSCTL_UINT(_net_inet_tcp_cc_hystartplusplus, OID_AUTO, n_rttsamples, + CTLFLAG_RW, + &hystart_n_rttsamples, 8, + "The number of RTT samples that must be seen to consider HyStart++"); + +SYSCTL_UINT(_net_inet_tcp_cc_hystartplusplus, OID_AUTO, css_growth_div, + CTLFLAG_RW, + &hystart_css_growth_div, 4, + "The divisor to the growth when in Hystart++ CSS"); + +SYSCTL_UINT(_net_inet_tcp_cc_hystartplusplus, OID_AUTO, css_rounds, + CTLFLAG_RW, + &hystart_css_rounds, 5, + "The number of rounds HyStart++ lasts in CSS before falling to CA"); + +SYSCTL_UINT(_net_inet_tcp_cc_hystartplusplus, OID_AUTO, bblogs, + CTLFLAG_RW, + &hystart_bblogs, 0, + "Do we enable HyStart++ Black Box logs to be generated if BB logging is on"); + VNET_DEFINE(int, cc_do_abe) = 0; SYSCTL_INT(_net_inet_tcp_cc, OID_AUTO, abe, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(cc_do_abe), 0, diff --git a/sys/netinet/cc/cc.h b/sys/netinet/cc/cc.h index 4282f0585bbd..6f942da7aa83 100644 --- a/sys/netinet/cc/cc.h +++ b/sys/netinet/cc/cc.h @@ -72,6 +72,15 @@ VNET_DECLARE(int, cc_abe_frlossreduce); /* Define the new net.inet.tcp.cc sysctl tree. */ SYSCTL_DECL(_net_inet_tcp_cc); +/* For CC modules that use hystart++ */ +extern uint32_t hystart_lowcwnd; +extern uint32_t hystart_minrtt_thresh; +extern uint32_t hystart_maxrtt_thresh; +extern uint32_t hystart_n_rttsamples; +extern uint32_t hystart_css_growth_div; +extern uint32_t hystart_css_rounds; +extern uint32_t hystart_bblogs; + /* CC housekeeping functions. */ int cc_register_algo(struct cc_algo *add_cc); int cc_deregister_algo(struct cc_algo *remove_cc); @@ -106,6 +115,9 @@ struct cc_var { #define CCF_CHG_MAX_CWND 0x0080 /* Cubic max_cwnd changed, for K */ #define CCF_USR_IWND 0x0100 /* User specified initial window */ #define CCF_USR_IWND_INIT_NSEG 0x0200 /* Convert segs to bytes on conn init */ +#define CCF_HYSTART_ALLOWED 0x0400 /* If the CC supports it Hystart is allowed */ +#define CCF_HYSTART_CAN_SH_CWND 0x0800 /* Can hystart when going CSS -> CA slam the cwnd */ +#define CCF_HYSTART_CONS_SSTH 0x1000 /* Should hystart use the more conservative ssthresh */ /* ACK types passed to the ack_received() hook. */ #define CC_ACK 0x0001 /* Regular in sequence ACK. */ diff --git a/sys/netinet/cc/cc_cubic.c b/sys/netinet/cc/cc_cubic.c index a4146423b0ae..e7d86a25156a 100644 --- a/sys/netinet/cc/cc_cubic.c +++ b/sys/netinet/cc/cc_cubic.c @@ -70,6 +70,8 @@ __FBSDID("$FreeBSD$"); #include <netinet/tcp_seq.h> #include <netinet/tcp_timer.h> #include <netinet/tcp_var.h> +#include <netinet/tcp_log_buf.h> +#include <netinet/tcp_hpts.h> #include <netinet/cc/cc.h> #include <netinet/cc/cc_cubic.h> #include <netinet/cc/cc_module.h> @@ -85,39 +87,9 @@ static void cubic_record_rtt(struct cc_var *ccv); static void cubic_ssthresh_update(struct cc_var *ccv, uint32_t maxseg); static void cubic_after_idle(struct cc_var *ccv); static size_t cubic_data_sz(void); - -struct cubic { - /* Cubic K in fixed point form with CUBIC_SHIFT worth of precision. */ - int64_t K; - /* Sum of RTT samples across an epoch in ticks. */ - int64_t sum_rtt_ticks; - /* cwnd at the most recent congestion event. */ - unsigned long max_cwnd; - /* cwnd at the previous congestion event. */ - unsigned long prev_max_cwnd; - /* A copy of prev_max_cwnd. Used for CC_RTO_ERR */ - unsigned long prev_max_cwnd_cp; - /* various flags */ - uint32_t flags; -#define CUBICFLAG_CONG_EVENT 0x00000001 /* congestion experienced */ -#define CUBICFLAG_IN_SLOWSTART 0x00000002 /* in slow start */ -#define CUBICFLAG_IN_APPLIMIT 0x00000004 /* application limited */ -#define CUBICFLAG_RTO_EVENT 0x00000008 /* RTO experienced */ - /* Minimum observed rtt in ticks. */ - int min_rtt_ticks; - /* Mean observed rtt between congestion epochs. */ - int mean_rtt_ticks; - /* ACKs since last congestion event. */ - int epoch_ack_count; - /* Timestamp (in ticks) of arriving in congestion avoidance from last - * congestion event. - */ - int t_last_cong; - /* Timestamp (in ticks) of a previous congestion event. Used for - * CC_RTO_ERR. - */ - int t_last_cong_prev; -}; +static void cubic_newround(struct cc_var *ccv, uint32_t round_cnt); +static void cubic_rttsample(struct cc_var *ccv, uint32_t usec_rtt, + uint32_t rxtcnt, uint32_t fas); struct cc_algo cubic_cc_algo = { .name = "cubic", @@ -129,9 +101,140 @@ struct cc_algo cubic_cc_algo = { .mod_init = cubic_mod_init, .post_recovery = cubic_post_recovery, .after_idle = cubic_after_idle, - .cc_data_sz = cubic_data_sz + .cc_data_sz = cubic_data_sz, + .rttsample = cubic_rttsample, + .newround = cubic_newround }; +static void +cubic_log_hystart_event(struct cc_var *ccv, struct cubic *cubicd, uint8_t mod, uint32_t flex1) +{ + /* + * Types of logs (mod value) + * 1 - rtt_thresh in flex1, checking to see if RTT is to great. + * 2 - rtt is too great, rtt_thresh in flex1. + * 3 - CSS is active incr in flex1 + * 4 - A new round is beginning flex1 is round count + * 5 - A new RTT measurement flex1 is the new measurement. + * 6 - We enter CA ssthresh is also in flex1. + * 7 - Socket option to change hystart executed opt.val in flex1. + * 8 - Back out of CSS into SS, flex1 is the css_baseline_minrtt + * 9 - We enter CA, via an ECN mark. + * 10 - We enter CA, via a loss. + * 11 - We have slipped out of SS into CA via cwnd growth. + * 12 - After idle has re-enabled hystart++ + */ + struct tcpcb *tp; + + if (hystart_bblogs == 0) + return; + tp = ccv->ccvc.tcp; + if (tp->t_logstate != TCP_LOG_STATE_OFF) { + union tcp_log_stackspecific log; + struct timeval tv; + + memset(&log, 0, sizeof(log)); + log.u_bbr.flex1 = flex1; + log.u_bbr.flex2 = cubicd->css_current_round_minrtt; + log.u_bbr.flex3 = cubicd->css_lastround_minrtt; + log.u_bbr.flex4 = cubicd->css_rttsample_count; + log.u_bbr.flex5 = cubicd->css_entered_at_round; + log.u_bbr.flex6 = cubicd->css_baseline_minrtt; + /* We only need bottom 16 bits of flags */ + log.u_bbr.flex7 = cubicd->flags & 0x0000ffff; + log.u_bbr.flex8 = mod; + log.u_bbr.epoch = cubicd->css_current_round; + log.u_bbr.timeStamp = tcp_get_usecs(&tv); + log.u_bbr.lt_epoch = cubicd->css_fas_at_css_entry; + log.u_bbr.pkts_out = cubicd->css_last_fas; + log.u_bbr.delivered = cubicd->css_lowrtt_fas; + log.u_bbr.pkt_epoch = ccv->flags; + TCP_LOG_EVENTP(tp, NULL, + &tp->t_inpcb->inp_socket->so_rcv, + &tp->t_inpcb->inp_socket->so_snd, + TCP_HYSTART, 0, + 0, &log, false, &tv); + } +} + +static void +cubic_does_slow_start(struct cc_var *ccv, struct cubic *cubicd) +{ + /* + * In slow-start with ABC enabled and no RTO in sight? + * (Must not use abc_l_var > 1 if slow starting after + * an RTO. On RTO, snd_nxt = snd_una, so the + * snd_nxt == snd_max check is sufficient to + * handle this). + * + * XXXLAS: Find a way to signal SS after RTO that + * doesn't rely on tcpcb vars. + */ + u_int cw = CCV(ccv, snd_cwnd); + u_int incr = CCV(ccv, t_maxseg); + uint16_t abc_val; + + cubicd->flags |= CUBICFLAG_IN_SLOWSTART; + if (ccv->flags & CCF_USE_LOCAL_ABC) + abc_val = ccv->labc; + else + abc_val = V_tcp_abc_l_var; + if ((ccv->flags & CCF_HYSTART_ALLOWED) && + (cubicd->flags & CUBICFLAG_HYSTART_ENABLED) && + ((cubicd->flags & CUBICFLAG_HYSTART_IN_CSS) == 0)) { + /* + * Hystart is allowed and still enabled and we are not yet + * in CSS. Lets check to see if we can make a decision on + * if we need to go into CSS. + */ + if ((cubicd->css_rttsample_count >= hystart_n_rttsamples) && + (cubicd->css_current_round_minrtt != 0xffffffff) && + (cubicd->css_lastround_minrtt != 0xffffffff)) { + uint32_t rtt_thresh; + + /* Clamp (minrtt_thresh, lastround/8, maxrtt_thresh) */ + rtt_thresh = (cubicd->css_lastround_minrtt >> 3); + if (rtt_thresh < hystart_minrtt_thresh) + rtt_thresh = hystart_minrtt_thresh; + if (rtt_thresh > hystart_maxrtt_thresh) + rtt_thresh = hystart_maxrtt_thresh; + cubic_log_hystart_event(ccv, cubicd, 1, rtt_thresh); + + if (cubicd->css_current_round_minrtt >= (cubicd->css_lastround_minrtt + rtt_thresh)) { + /* Enter CSS */ + cubicd->flags |= CUBICFLAG_HYSTART_IN_CSS; + cubicd->css_fas_at_css_entry = cubicd->css_lowrtt_fas; + /* + * The draft (v4) calls for us to set baseline to css_current_round_min + * but that can cause an oscillation. We probably shoudl be using + * css_lastround_minrtt, but the authors insist that will cause + * issues on exiting early. We will leave the draft version for now + * but I suspect this is incorrect. + */ + cubicd->css_baseline_minrtt = cubicd->css_current_round_minrtt; + cubicd->css_entered_at_round = cubicd->css_current_round; + cubic_log_hystart_event(ccv, cubicd, 2, rtt_thresh); + } + } + } + if (CCV(ccv, snd_nxt) == CCV(ccv, snd_max)) + incr = min(ccv->bytes_this_ack, + ccv->nsegs * abc_val * + CCV(ccv, t_maxseg)); + else + incr = min(ccv->bytes_this_ack, CCV(ccv, t_maxseg)); + + /* Only if Hystart is enabled will the flag get set */ + if (cubicd->flags & CUBICFLAG_HYSTART_IN_CSS) { + incr /= hystart_css_growth_div; + cubic_log_hystart_event(ccv, cubicd, 3, incr); + } + /* ABC is on by default, so incr equals 0 frequently. */ + if (incr > 0) + CCV(ccv, snd_cwnd) = min((cw + incr), + TCP_MAXWIN << CCV(ccv, snd_scale)); +} + static void cubic_ack_received(struct cc_var *ccv, uint16_t type) { @@ -151,9 +254,19 @@ cubic_ack_received(struct cc_var *ccv, uint16_t type) /* Use the logic in NewReno ack_received() for slow start. */ if (CCV(ccv, snd_cwnd) <= CCV(ccv, snd_ssthresh) || cubic_data->min_rtt_ticks == TCPTV_SRTTBASE) { - cubic_data->flags |= CUBICFLAG_IN_SLOWSTART; - newreno_cc_ack_received(ccv, type); + cubic_does_slow_start(ccv, cubic_data); } else { + if (cubic_data->flags & CUBICFLAG_HYSTART_IN_CSS) { + /* + * We have slipped into CA with + * CSS active. Deactivate all. + */ + /* Turn off the CSS flag */ + cubic_data->flags &= ~CUBICFLAG_HYSTART_IN_CSS; + /* Disable use of CSS in the future except long idle */ + cubic_data->flags &= ~CUBICFLAG_HYSTART_ENABLED; + cubic_log_hystart_event(ccv, cubic_data, 11, CCV(ccv, snd_ssthresh)); + } if ((cubic_data->flags & CUBICFLAG_RTO_EVENT) && (cubic_data->flags & CUBICFLAG_IN_SLOWSTART)) { /* RFC8312 Section 4.7 */ @@ -245,7 +358,14 @@ cubic_after_idle(struct cc_var *ccv) cubic_data->max_cwnd = ulmax(cubic_data->max_cwnd, CCV(ccv, snd_cwnd)); cubic_data->K = cubic_k(cubic_data->max_cwnd / CCV(ccv, t_maxseg)); - + if ((cubic_data->flags & CUBICFLAG_HYSTART_ENABLED) == 0) { + /* + * Re-enable hystart if we have been idle. + */ + cubic_data->flags &= ~CUBICFLAG_HYSTART_IN_CSS; + cubic_data->flags |= CUBICFLAG_HYSTART_ENABLED; + cubic_log_hystart_event(ccv, cubic_data, 12, CCV(ccv, snd_ssthresh)); + } newreno_cc_after_idle(ccv); cubic_data->t_last_cong = ticks; } @@ -281,6 +401,17 @@ cubic_cb_init(struct cc_var *ccv, void *ptr) cubic_data->mean_rtt_ticks = 1; ccv->cc_data = cubic_data; + cubic_data->flags = CUBICFLAG_HYSTART_ENABLED; + /* At init set both to infinity */ + cubic_data->css_lastround_minrtt = 0xffffffff; + cubic_data->css_current_round_minrtt = 0xffffffff; + cubic_data->css_current_round = 0; + cubic_data->css_baseline_minrtt = 0xffffffff; + cubic_data->css_rttsample_count = 0; + cubic_data->css_entered_at_round = 0; + cubic_data->css_fas_at_css_entry = 0; + cubic_data->css_lowrtt_fas = 0; + cubic_data->css_last_fas = 0; return (0); } @@ -299,6 +430,12 @@ cubic_cong_signal(struct cc_var *ccv, uint32_t type) switch (type) { case CC_NDUPACK: + if (cubic_data->flags & CUBICFLAG_HYSTART_ENABLED) { + /* Make sure the flags are all off we had a loss */ + cubic_data->flags &= ~CUBICFLAG_HYSTART_ENABLED; + cubic_data->flags &= ~CUBICFLAG_HYSTART_IN_CSS; + cubic_log_hystart_event(ccv, cubic_data, 10, CCV(ccv, snd_ssthresh)); + } if (!IN_FASTRECOVERY(CCV(ccv, t_flags))) { if (!IN_CONGRECOVERY(CCV(ccv, t_flags))) { cubic_ssthresh_update(ccv, mss); @@ -311,6 +448,12 @@ cubic_cong_signal(struct cc_var *ccv, uint32_t type) break; case CC_ECN: + if (cubic_data->flags & CUBICFLAG_HYSTART_ENABLED) { + /* Make sure the flags are all off we had a loss */ + cubic_data->flags &= ~CUBICFLAG_HYSTART_ENABLED; + cubic_data->flags &= ~CUBICFLAG_HYSTART_IN_CSS; + cubic_log_hystart_event(ccv, cubic_data, 9, CCV(ccv, snd_ssthresh)); + } if (!IN_CONGRECOVERY(CCV(ccv, t_flags))) { cubic_ssthresh_update(ccv, mss); cubic_data->flags |= CUBICFLAG_CONG_EVENT; @@ -495,5 +638,79 @@ cubic_ssthresh_update(struct cc_var *ccv, uint32_t maxseg) CCV(ccv, snd_ssthresh) = max(ssthresh, 2 * maxseg); } +static void +cubic_rttsample(struct cc_var *ccv, uint32_t usec_rtt, uint32_t rxtcnt, uint32_t fas) +{ + struct cubic *cubicd; + + cubicd = ccv->cc_data; + if (rxtcnt > 1) { + /* + * Only look at RTT's that are non-ambiguous. + */ + return; + } + cubicd->css_rttsample_count++; + cubicd->css_last_fas = fas; + if (cubicd->css_current_round_minrtt > usec_rtt) { + cubicd->css_current_round_minrtt = usec_rtt; + cubicd->css_lowrtt_fas = cubicd->css_last_fas; + } + if ((cubicd->css_rttsample_count >= hystart_n_rttsamples) && + (cubicd->css_current_round_minrtt != 0xffffffff) && + (cubicd->css_lastround_minrtt != 0xffffffff)) { + /* + * We were in CSS and the RTT is now less, we + * entered CSS erroneously. + */ + cubicd->flags &= ~CUBICFLAG_HYSTART_IN_CSS; + cubic_log_hystart_event(ccv, cubicd, 8, cubicd->css_baseline_minrtt); + cubicd->css_baseline_minrtt = 0xffffffff; + } + if (cubicd->flags & CUBICFLAG_HYSTART_ENABLED) + cubic_log_hystart_event(ccv, cubicd, 5, usec_rtt); +} + +static void +cubic_newround(struct cc_var *ccv, uint32_t round_cnt) +{ + struct cubic *cubicd; + + cubicd = ccv->cc_data; + /* We have entered a new round */ + cubicd->css_lastround_minrtt = cubicd->css_current_round_minrtt; + cubicd->css_current_round_minrtt = 0xffffffff; + cubicd->css_rttsample_count = 0; + cubicd->css_current_round = round_cnt; + if ((cubicd->flags & CUBICFLAG_HYSTART_IN_CSS) && + ((round_cnt - cubicd->css_entered_at_round) >= hystart_css_rounds)) { + /* Enter CA */ + if (ccv->flags & CCF_HYSTART_CAN_SH_CWND) { + /* + * We engage more than snd_ssthresh, engage + * the brakes!! Though we will stay in SS to + * creep back up again, so lets leave CSS active + * and give us hystart_css_rounds more rounds. + */ + if (ccv->flags & CCF_HYSTART_CONS_SSTH) { + CCV(ccv, snd_ssthresh) = ((cubicd->css_lowrtt_fas + cubicd->css_fas_at_css_entry) / 2); + } else { + CCV(ccv, snd_ssthresh) = cubicd->css_lowrtt_fas; + } + CCV(ccv, snd_cwnd) = cubicd->css_fas_at_css_entry; + cubicd->css_entered_at_round = round_cnt; + } else { + CCV(ccv, snd_ssthresh) = CCV(ccv, snd_cwnd); + /* Turn off the CSS flag */ + cubicd->flags &= ~CUBICFLAG_HYSTART_IN_CSS; + /* Disable use of CSS in the future except long idle */ + cubicd->flags &= ~CUBICFLAG_HYSTART_ENABLED; + } + cubic_log_hystart_event(ccv, cubicd, 6, CCV(ccv, snd_ssthresh)); + } + if (cubicd->flags & CUBICFLAG_HYSTART_ENABLED) + cubic_log_hystart_event(ccv, cubicd, 4, round_cnt); +} + DECLARE_CC_MODULE(cubic, &cubic_cc_algo); MODULE_VERSION(cubic, 2); diff --git a/sys/netinet/cc/cc_cubic.h b/sys/netinet/cc/cc_cubic.h index 13681aae793c..2042590182fc 100644 --- a/sys/netinet/cc/cc_cubic.h +++ b/sys/netinet/cc/cc_cubic.h @@ -78,6 +78,55 @@ */ #define CUBED_ROOT_MAX_ULONG 448845 +/* Flags used in the cubic structure */ +#define CUBICFLAG_CONG_EVENT 0x00000001 /* congestion experienced */ +#define CUBICFLAG_IN_SLOWSTART 0x00000002 /* in slow start */ +#define CUBICFLAG_IN_APPLIMIT 0x00000004 /* application limited */ +#define CUBICFLAG_RTO_EVENT 0x00000008 /* RTO experienced */ +#define CUBICFLAG_HYSTART_ENABLED 0x00000010 /* Hystart++ is enabled */ +#define CUBICFLAG_HYSTART_IN_CSS 0x00000020 /* We are in Hystart++ CSS */ + +/* Kernel only bits */ +#ifdef _KERNEL +struct cubic { + /* Cubic K in fixed point form with CUBIC_SHIFT worth of precision. */ + int64_t K; + /* Sum of RTT samples across an epoch in ticks. */ + int64_t sum_rtt_ticks; + /* cwnd at the most recent congestion event. */ + unsigned long max_cwnd; + /* cwnd at the previous congestion event. */ + unsigned long prev_max_cwnd; + /* A copy of prev_max_cwnd. Used for CC_RTO_ERR */ + unsigned long prev_max_cwnd_cp; + /* various flags */ + uint32_t flags; + /* Minimum observed rtt in ticks. */ + int min_rtt_ticks; + /* Mean observed rtt between congestion epochs. */ + int mean_rtt_ticks; + /* ACKs since last congestion event. */ + int epoch_ack_count; + /* Timestamp (in ticks) of arriving in congestion avoidance from last + * congestion event. + */ + int t_last_cong; + /* Timestamp (in ticks) of a previous congestion event. Used for + * CC_RTO_ERR. + */ + int t_last_cong_prev; + uint32_t css_baseline_minrtt; + uint32_t css_current_round_minrtt; + uint32_t css_lastround_minrtt; + uint32_t css_rttsample_count; + uint32_t css_entered_at_round; + uint32_t css_current_round; + uint32_t css_fas_at_css_entry; + uint32_t css_lowrtt_fas; + uint32_t css_last_fas; +}; +#endif + /* Userland only bits. */ #ifndef _KERNEL diff --git a/sys/netinet/cc/cc_newreno.c b/sys/netinet/cc/cc_newreno.c index 3280e270ede3..06b53d76d8e9 100644 --- a/sys/netinet/cc/cc_newreno.c +++ b/sys/netinet/cc/cc_newreno.c @@ -116,14 +116,6 @@ struct cc_algo newreno_cc_algo = { .cc_data_sz = newreno_data_sz, }; -static uint32_t hystart_lowcwnd = 16; -static uint32_t hystart_minrtt_thresh = 4000; -static uint32_t hystart_maxrtt_thresh = 16000; -static uint32_t hystart_n_rttsamples = 8; -static uint32_t hystart_css_growth_div = 4; -static uint32_t hystart_css_rounds = 5; -static uint32_t hystart_bblogs = 0; - static void newreno_log_hystart_event(struct cc_var *ccv, struct newreno *nreno, uint8_t mod, uint32_t flex1) { @@ -137,6 +129,10 @@ newreno_log_hystart_event(struct cc_var *ccv, struct newreno *nreno, uint8_t mod * 6 - We enter CA ssthresh is also in flex1. * 7 - Socket option to change hystart executed opt.val in flex1. * 8 - Back out of CSS into SS, flex1 is the css_baseline_minrtt + * 9 - We enter CA, via an ECN mark. + * 10 - We enter CA, via a loss. + * 11 - We have slipped out of SS into CA via cwnd growth. + * 12 - After idle has re-enabled hystart++ */ struct tcpcb *tp; @@ -162,6 +158,7 @@ newreno_log_hystart_event(struct cc_var *ccv, struct newreno *nreno, uint8_t mod log.u_bbr.lt_epoch = nreno->css_fas_at_css_entry; log.u_bbr.pkts_out = nreno->css_last_fas; log.u_bbr.delivered = nreno->css_lowrtt_fas; + log.u_bbr.pkt_epoch = ccv->flags; TCP_LOG_EVENTP(tp, NULL, &tp->t_inpcb->inp_socket->so_rcv, &tp->t_inpcb->inp_socket->so_snd, @@ -265,6 +262,7 @@ newreno_ack_received(struct cc_var *ccv, uint16_t type) nreno->newreno_flags &= ~CC_NEWRENO_HYSTART_IN_CSS; /* Disable use of CSS in the future except long idle */ nreno->newreno_flags &= ~CC_NEWRENO_HYSTART_ENABLED; + newreno_log_hystart_event(ccv, nreno, 11, CCV(ccv, snd_ssthresh)); } if (V_tcp_do_rfc3465) { if (ccv->flags & CCF_ABC_SENTAWND) @@ -290,7 +288,7 @@ newreno_ack_received(struct cc_var *ccv, uint16_t type) abc_val = ccv->labc; else abc_val = V_tcp_abc_l_var; - if ((nreno->newreno_flags & CC_NEWRENO_HYSTART_ALLOWED) && + if ((ccv->flags & CCF_HYSTART_ALLOWED) && (nreno->newreno_flags & CC_NEWRENO_HYSTART_ENABLED) && ((nreno->newreno_flags & CC_NEWRENO_HYSTART_IN_CSS) == 0)) { /* @@ -299,8 +297,8 @@ newreno_ack_received(struct cc_var *ccv, uint16_t type) * if we need to go into CSS. */ if ((nreno->css_rttsample_count >= hystart_n_rttsamples) && - (CCV(ccv, snd_cwnd) > - (hystart_lowcwnd * tcp_fixed_maxseg(ccv->ccvc.tcp)))) { + (nreno->css_current_round_minrtt != 0xffffffff) && + (nreno->css_lastround_minrtt != 0xffffffff)) { uint32_t rtt_thresh; /* Clamp (minrtt_thresh, lastround/8, maxrtt_thresh) */ @@ -314,6 +312,13 @@ newreno_ack_received(struct cc_var *ccv, uint16_t type) /* Enter CSS */ nreno->newreno_flags |= CC_NEWRENO_HYSTART_IN_CSS; nreno->css_fas_at_css_entry = nreno->css_lowrtt_fas; + /* + * The draft (v4) calls for us to set baseline to css_current_round_min + * but that can cause an oscillation. We probably shoudl be using + * css_lastround_minrtt, but the authors insist that will cause + * issues on exiting early. We will leave the draft version for now + * but I suspect this is incorrect. + */ nreno->css_baseline_minrtt = nreno->css_current_round_minrtt; nreno->css_entered_at_round = nreno->css_current_round; newreno_log_hystart_event(ccv, nreno, 2, rtt_thresh); @@ -348,14 +353,12 @@ newreno_after_idle(struct cc_var *ccv) nreno = ccv->cc_data; newreno_cc_after_idle(ccv); if ((nreno->newreno_flags & CC_NEWRENO_HYSTART_ENABLED) == 0) { - if (CCV(ccv, snd_cwnd) <= (hystart_lowcwnd * tcp_fixed_maxseg(ccv->ccvc.tcp))) { - /* - * Re-enable hystart if our cwnd has fallen below - * the hystart lowcwnd point. - */ - nreno->newreno_flags &= ~CC_NEWRENO_HYSTART_IN_CSS; - nreno->newreno_flags |= CC_NEWRENO_HYSTART_ENABLED; - } + /* + * Re-enable hystart if we have been idle. + */ + nreno->newreno_flags &= ~CC_NEWRENO_HYSTART_IN_CSS; + nreno->newreno_flags |= CC_NEWRENO_HYSTART_ENABLED; + newreno_log_hystart_event(ccv, nreno, 12, CCV(ccv, snd_ssthresh)); } } @@ -400,6 +403,7 @@ newreno_cong_signal(struct cc_var *ccv, uint32_t type) /* Make sure the flags are all off we had a loss */ nreno->newreno_flags &= ~CC_NEWRENO_HYSTART_ENABLED; nreno->newreno_flags &= ~CC_NEWRENO_HYSTART_IN_CSS; + newreno_log_hystart_event(ccv, nreno, 10, CCV(ccv, snd_ssthresh)); } if (!IN_FASTRECOVERY(CCV(ccv, t_flags))) { if (IN_CONGRECOVERY(CCV(ccv, t_flags) && @@ -418,6 +422,7 @@ newreno_cong_signal(struct cc_var *ccv, uint32_t type) /* Make sure the flags are all off we had a loss */ nreno->newreno_flags &= ~CC_NEWRENO_HYSTART_ENABLED; nreno->newreno_flags &= ~CC_NEWRENO_HYSTART_IN_CSS; + newreno_log_hystart_event(ccv, nreno, 9, CCV(ccv, snd_ssthresh)); } if (!IN_CONGRECOVERY(CCV(ccv, t_flags))) { CCV(ccv, snd_ssthresh) = cwin; @@ -460,18 +465,6 @@ newreno_ctl_output(struct cc_var *ccv, struct sockopt *sopt, void *buf) nreno->beta_ecn = opt->val; nreno->newreno_flags |= CC_NEWRENO_BETA_ECN_ENABLED; break; - case CC_NEWRENO_ENABLE_HYSTART: - /* Allow hystart on this connection */ - if (opt->val != 0) { - nreno->newreno_flags |= CC_NEWRENO_HYSTART_ALLOWED; - if (opt->val > 1) - nreno->newreno_flags |= CC_NEWRENO_HYSTART_CAN_SH_CWND; - if (opt->val > 2) - nreno->newreno_flags |= CC_NEWRENO_HYSTART_CONS_SSTH; - } else - nreno->newreno_flags &= ~(CC_NEWRENO_HYSTART_ALLOWED|CC_NEWRENO_HYSTART_CAN_SH_CWND|CC_NEWRENO_HYSTART_CONS_SSTH); - newreno_log_hystart_event(ccv, nreno, 7, opt->val); - break; default: return (ENOPROTOOPT); } @@ -486,17 +479,6 @@ newreno_ctl_output(struct cc_var *ccv, struct sockopt *sopt, void *buf) opt->val = (nreno == NULL) ? V_newreno_beta_ecn : nreno->beta_ecn; break; - case CC_NEWRENO_ENABLE_HYSTART: - if (nreno->newreno_flags & CC_NEWRENO_HYSTART_ALLOWED) { - if (nreno->newreno_flags & CC_NEWRENO_HYSTART_CONS_SSTH) - opt->val = 3; - else if (nreno->newreno_flags & CC_NEWRENO_HYSTART_CAN_SH_CWND) - opt->val = 2; - else - opt->val = 1; - } else - opt->val = 0; - break; default: return (ENOPROTOOPT); } @@ -542,14 +524,14 @@ newreno_newround(struct cc_var *ccv, uint32_t round_cnt) if ((nreno->newreno_flags & CC_NEWRENO_HYSTART_IN_CSS) && ((round_cnt - nreno->css_entered_at_round) >= hystart_css_rounds)) { /* Enter CA */ - if (nreno->newreno_flags & CC_NEWRENO_HYSTART_CAN_SH_CWND) { + if (ccv->flags & CCF_HYSTART_CAN_SH_CWND) { /* * We engage more than snd_ssthresh, engage * the brakes!! Though we will stay in SS to * creep back up again, so lets leave CSS active * and give us hystart_css_rounds more rounds. */ - if (nreno->newreno_flags & CC_NEWRENO_HYSTART_CONS_SSTH) { + if (ccv->flags & CCF_HYSTART_CONS_SSTH) { CCV(ccv, snd_ssthresh) = ((nreno->css_lowrtt_fas + nreno->css_fas_at_css_entry) / 2); } else { CCV(ccv, snd_ssthresh) = nreno->css_lowrtt_fas; @@ -565,7 +547,8 @@ newreno_newround(struct cc_var *ccv, uint32_t round_cnt) } newreno_log_hystart_event(ccv, nreno, 6, CCV(ccv, snd_ssthresh)); } - newreno_log_hystart_event(ccv, nreno, 4, round_cnt); + if (nreno->newreno_flags & CC_NEWRENO_HYSTART_ENABLED) + newreno_log_hystart_event(ccv, nreno, 4, round_cnt); } static void @@ -586,9 +569,9 @@ newreno_rttsample(struct cc_var *ccv, uint32_t usec_rtt, uint32_t rxtcnt, uint32 nreno->css_current_round_minrtt = usec_rtt; nreno->css_lowrtt_fas = nreno->css_last_fas; } - if ((nreno->newreno_flags & CC_NEWRENO_HYSTART_IN_CSS) && - (nreno->css_rttsample_count >= hystart_n_rttsamples) && - (nreno->css_baseline_minrtt > nreno->css_current_round_minrtt)) { + if ((nreno->css_rttsample_count >= hystart_n_rttsamples) && + (nreno->css_current_round_minrtt != 0xffffffff) && + (nreno->css_lastround_minrtt != 0xffffffff)) { /* * We were in CSS and the RTT is now less, we * entered CSS erroneously. @@ -597,7 +580,8 @@ newreno_rttsample(struct cc_var *ccv, uint32_t usec_rtt, uint32_t rxtcnt, uint32 newreno_log_hystart_event(ccv, nreno, 8, nreno->css_baseline_minrtt); nreno->css_baseline_minrtt = 0xffffffff; } - newreno_log_hystart_event(ccv, nreno, 5, usec_rtt); + if (nreno->newreno_flags & CC_NEWRENO_HYSTART_ENABLED) + newreno_log_hystart_event(ccv, nreno, 5, usec_rtt); } SYSCTL_DECL(_net_inet_tcp_cc_newreno); @@ -615,45 +599,5 @@ SYSCTL_PROC(_net_inet_tcp_cc_newreno, OID_AUTO, beta_ecn, &VNET_NAME(newreno_beta_ecn), 3, &newreno_beta_handler, "IU", "New Reno beta ecn, specified as number between 1 and 100"); -SYSCTL_NODE(_net_inet_tcp_cc_newreno, OID_AUTO, hystartplusplus, - CTLFLAG_RW | CTLFLAG_MPSAFE, NULL, - "New Reno related HyStart++ settings"); - -SYSCTL_UINT(_net_inet_tcp_cc_newreno_hystartplusplus, OID_AUTO, lowcwnd, - CTLFLAG_RW, - &hystart_lowcwnd, 16, - "The number of MSS in the CWND before HyStart++ is active"); - -SYSCTL_UINT(_net_inet_tcp_cc_newreno_hystartplusplus, OID_AUTO, minrtt_thresh, - CTLFLAG_RW, - &hystart_minrtt_thresh, 4000, - "HyStarts++ minimum RTT thresh used in clamp (in microseconds)"); - -SYSCTL_UINT(_net_inet_tcp_cc_newreno_hystartplusplus, OID_AUTO, maxrtt_thresh, - CTLFLAG_RW, - &hystart_maxrtt_thresh, 16000, - "HyStarts++ maximum RTT thresh used in clamp (in microseconds)"); - -SYSCTL_UINT(_net_inet_tcp_cc_newreno_hystartplusplus, OID_AUTO, n_rttsamples, - CTLFLAG_RW, - &hystart_n_rttsamples, 8, - "The number of RTT samples that must be seen to consider HyStart++"); - -SYSCTL_UINT(_net_inet_tcp_cc_newreno_hystartplusplus, OID_AUTO, css_growth_div, - CTLFLAG_RW, - &hystart_css_growth_div, 4, - "The divisor to the growth when in Hystart++ CSS"); - -SYSCTL_UINT(_net_inet_tcp_cc_newreno_hystartplusplus, OID_AUTO, css_rounds, - CTLFLAG_RW, - &hystart_css_rounds, 5, - "The number of rounds HyStart++ lasts in CSS before falling to CA"); - -SYSCTL_UINT(_net_inet_tcp_cc_newreno_hystartplusplus, OID_AUTO, bblogs, - CTLFLAG_RW, - &hystart_bblogs, 0, - "Do we enable HyStart++ Black Box logs to be generated if BB logging is on"); - - DECLARE_CC_MODULE(newreno, &newreno_cc_algo); MODULE_VERSION(newreno, 2); diff --git a/sys/netinet/cc/cc_newreno.h b/sys/netinet/cc/cc_newreno.h index 42c41c7db287..791b64dfdc00 100644 --- a/sys/netinet/cc/cc_newreno.h +++ b/sys/netinet/cc/cc_newreno.h @@ -53,13 +53,9 @@ struct cc_newreno_opts { #define CC_NEWRENO_BETA 1 /* Beta for normal DUP-ACK/Sack recovery */ #define CC_NEWRENO_BETA_ECN 2 /* ECN Beta for Abe */ -#define CC_NEWRENO_ENABLE_HYSTART 3 /* Enable hystart */ /* Flags values */ -#define CC_NEWRENO_HYSTART_ALLOWED 0x0001 /* Does the tcp connection allow hystart? */ #define CC_NEWRENO_HYSTART_ENABLED 0x0002 /* We can do hystart, a loss removes this flag */ #define CC_NEWRENO_HYSTART_IN_CSS 0x0004 /* If we enter hystart CSS this flag is set */ -#define CC_NEWRENO_HYSTART_CAN_SH_CWND 0x0008 /* Can hystart when going CSS -> CA slam the cwnd */ -#define CC_NEWRENO_HYSTART_CONS_SSTH 0x0010 /* Should hystart use the more conservative sstrhesh */ #define CC_NEWRENO_BETA_ECN_ENABLED 0x0020 #endif /* _CC_NEWRENO_H */ diff --git a/sys/netinet/tcp_stacks/rack.c b/sys/netinet/tcp_stacks/rack.c index 4ad6a12ee43b..459520a4a0bb 100644 --- a/sys/netinet/tcp_stacks/rack.c +++ b/sys/netinet/tcp_stacks/rack.c @@ -12858,15 +12858,11 @@ rack_init(struct tcpcb *tp) rack_convert_rtts(tp); tp->t_rttlow = TICKS_2_USEC(tp->t_rttlow); if (rack_do_hystart) { - struct sockopt sopt; - struct cc_newreno_opts opt; - - sopt.sopt_valsize = sizeof(struct cc_newreno_opts); - sopt.sopt_dir = SOPT_SET; - opt.name = CC_NEWRENO_ENABLE_HYSTART; - opt.val = rack_do_hystart; - if (CC_ALGO(tp)->ctl_output != NULL) - (void)CC_ALGO(tp)->ctl_output(tp->ccv, &sopt, &opt); + tp->ccv->flags |= CCF_HYSTART_ALLOWED; + if (rack_do_hystart > 1) + tp->ccv->flags |= CCF_HYSTART_CAN_SH_CWND; + if (rack_do_hystart > 2) + tp->ccv->flags |= CCF_HYSTART_CONS_SSTH; } if (rack_def_profile) rack_set_profile(rack, rack_def_profile); @@ -13515,7 +13511,6 @@ rack_handle_probe_response(struct tcp_rack *rack, uint32_t tiwin, uint32_t us_ct } } - static int rack_do_compressed_ack_processing(struct tcpcb *tp, struct socket *so, struct mbuf *m, int nxt_pkt, struct timeval *tv) { @@ -13766,7 +13761,29 @@ rack_do_compressed_ack_processing(struct tcpcb *tp, struct socket *so, struct mb (((ae->ack - high_seq) + segsiz - 1) / segsiz)); #endif high_seq = ae->ack; - if (SEQ_GEQ(high_seq, rack->r_ctl.roundends)) { + if (rack_verbose_logging && (rack->rc_tp->t_logstate != TCP_LOG_STATE_OFF)) { + union tcp_log_stackspecific log; + struct timeval tv; + + memset(&log.u_bbr, 0, sizeof(log.u_bbr)); + log.u_bbr.timeStamp = tcp_get_usecs(&tv); + log.u_bbr.flex1 = high_seq; + log.u_bbr.flex2 = rack->r_ctl.roundends; + log.u_bbr.flex3 = rack->r_ctl.current_round; + log.u_bbr.rttProp = (uint64_t)CC_ALGO(tp)->newround; + log.u_bbr.flex8 = 8; + tcp_log_event_(tp, NULL, NULL, NULL, BBR_LOG_CWND, 0, + 0, &log, false, NULL, NULL, 0, &tv); + } + /* + * The draft (v3) calls for us to use SEQ_GEQ, but that + * causes issues when we are just going app limited. Lets + * instead use SEQ_GT <or> where its equal but more data + * is outstanding. + */ + if ((SEQ_GT(high_seq, rack->r_ctl.roundends)) || + ((high_seq == rack->r_ctl.roundends) && + SEQ_GT(tp->snd_max, tp->snd_una))) { rack->r_ctl.current_round++; rack->r_ctl.roundends = tp->snd_max; if (CC_ALGO(tp)->newround != NULL) { @@ -14194,7 +14211,7 @@ rack_do_segment_nounlock(struct mbuf *m, struct tcphdr *th, struct socket *so, * us_cts - is the time that LRO or hardware actually got the packet in microseconds. */ uint32_t cts, us_cts, ms_cts; - uint32_t tiwin; *** 115 LINES SKIPPED ***
