Re: [RFT v4] printk: allow increasing the ring buffer depending on the number of CPUs
On Mon, Jun 16, 2014 at 06:32:53PM +0200, Petr Mládek wrote: > On Sat 2014-06-14 11:52:45, Luis R. Rodriguez wrote: > > From: "Luis R. Rodriguez" > > > > The default size of the ring buffer is too small for machines > > with a large amount of CPUs under heavy load. What ends up > > happening when debugging is the ring buffer overlaps and chews > > up old messages making debugging impossible unless the size is > > passed as a kernel parameter. An idle system upon boot up will > > on average spew out only about one or two extra lines but where > > this really matters is on heavy load and that will vary widely > > depending on the system and environment. > > > > There are mechanisms to help increase the kernel ring buffer > > for tracing through debugfs, and those interfaces even allow growing > > the kernel ring buffer per CPU. We also have a static value which > > can be passed upon boot. Relying on debugfs however is not ideal > > for production, and relying on the value passed upon bootup is > > can only used *after* an issue has creeped up. Instead of being > > reactive this adds a proactive measure which lets you scale the > > amount of contributions you'd expect to the kernel ring buffer > > under load by each CPU in the worst case scenario. > > > > We use num_possible_cpus() to avoid complexities which could be > > introduced by dynamically changing the ring buffer size at run > > time, num_possible_cpus() lets us use the upper limit on possible > > number of CPUs therefore avoiding having to deal with hotplugging > > CPUs on and off. This introduces the kernel configuration option > > LOG_CPU_MIN_BUF_SHIFT which is used to specify the maximum amount > > of contributions to the kernel ring buffer in the worst case before > > the kernel ring buffer flips over, the size is specified as a power > > of 2. The total amount of contributions made by each CPU must be > > greater than half of the default kernel ring buffer size > > (1 << LOG_BUF_SHIFT bytes) in order to trigger an increase upon > > bootup. For example if LOG_BUF_SHIFT is 18 (256 KB) you'd require at > > least 128 KB contributions by other CPUs in order to trigger an > > increase. With a LOG_CPU_BUF_SHIFT of 12 (4 KB) you'd require at > > least anything over > 64 possible CPUs to trigger an increase. If > > you had 128 possible CPUs your kernel buffer size would be: > > > >((1 << 18) + ((128 - 1) * (1 << 12))) / 1024 = 764 KB > > > > This value is ignored when "log_buf_len" kernel parameter is used > > as it forces the exact size of the ring buffer to an expected value. > > > > In order to use num_possible_cpus() we need to make this a late call > > on the init process but that also means we cannot share __init code, > > the late_setup_log_buf() provided is therefore very similar to the > > setup_log_buf() but just modified slightly, in particular since we're > > at late boot we can now also use kzalloc(). > > > > Cc: Michal Hocko > > Cc: Petr Mladek > > Cc: Andrew Morton > > Cc: Joe Perches > > Cc: Arun KS > > Cc: Kees Cook > > Cc: Davidlohr Bueso > > Cc: Chris Metcalf > > Cc: linux-kernel@vger.kernel.org > > Signed-off-by: Luis R. Rodriguez > > --- > > > > This doesn't kfree() at halt but do we need to do it ? > > > > Documentation/kernel-parameters.txt | 8 -- > > include/linux/printk.h | 5 > > init/Kconfig| 45 +++- > > init/main.c | 1 + > > kernel/printk/printk.c | 51 > > + > > 5 files changed, 107 insertions(+), 3 deletions(-) > > > > diff --git a/Documentation/kernel-parameters.txt > > b/Documentation/kernel-parameters.txt > > index 6eaa9cd..229d031 100644 > > --- a/Documentation/kernel-parameters.txt > > +++ b/Documentation/kernel-parameters.txt > > @@ -1685,8 +1685,12 @@ bytes respectively. Such letter suffixes can also be > > entirely omitted. > > 7 (KERN_DEBUG) debug-level messages > > > > log_buf_len=n[KMG] Sets the size of the printk ring buffer, > > - in bytes. n must be a power of two. The default > > - size is set in the kernel config file. > > + in bytes. n must be a power of two and greater > > + than the minimal size. The minimal size is defined > > + by LOG_BUF_SHIFT kernel config parameter. There is > > + also CONFIG_LOG_CPU_MIN_BUF_SHIFT config parameter > > + that allows to increase the default size depending on > > + the number of CPUs. See init/Kconfig for more details. > > > > logo.nologo [FB] Disables display of the built-in Linux logo. > > This may be used to provide more screen space for > > diff --git a/include/linux/printk.h b/include/linux/printk.h > > index 319ff7e..315a0b7 100644 > > --- a/include/linux/printk.h > > +++
Re: [RFT v4] printk: allow increasing the ring buffer depending on the number of CPUs
On Sat 2014-06-14 11:52:45, Luis R. Rodriguez wrote: > From: "Luis R. Rodriguez" > > The default size of the ring buffer is too small for machines > with a large amount of CPUs under heavy load. What ends up > happening when debugging is the ring buffer overlaps and chews > up old messages making debugging impossible unless the size is > passed as a kernel parameter. An idle system upon boot up will > on average spew out only about one or two extra lines but where > this really matters is on heavy load and that will vary widely > depending on the system and environment. > > There are mechanisms to help increase the kernel ring buffer > for tracing through debugfs, and those interfaces even allow growing > the kernel ring buffer per CPU. We also have a static value which > can be passed upon boot. Relying on debugfs however is not ideal > for production, and relying on the value passed upon bootup is > can only used *after* an issue has creeped up. Instead of being > reactive this adds a proactive measure which lets you scale the > amount of contributions you'd expect to the kernel ring buffer > under load by each CPU in the worst case scenario. > > We use num_possible_cpus() to avoid complexities which could be > introduced by dynamically changing the ring buffer size at run > time, num_possible_cpus() lets us use the upper limit on possible > number of CPUs therefore avoiding having to deal with hotplugging > CPUs on and off. This introduces the kernel configuration option > LOG_CPU_MIN_BUF_SHIFT which is used to specify the maximum amount > of contributions to the kernel ring buffer in the worst case before > the kernel ring buffer flips over, the size is specified as a power > of 2. The total amount of contributions made by each CPU must be > greater than half of the default kernel ring buffer size > (1 << LOG_BUF_SHIFT bytes) in order to trigger an increase upon > bootup. For example if LOG_BUF_SHIFT is 18 (256 KB) you'd require at > least 128 KB contributions by other CPUs in order to trigger an > increase. With a LOG_CPU_BUF_SHIFT of 12 (4 KB) you'd require at > least anything over > 64 possible CPUs to trigger an increase. If > you had 128 possible CPUs your kernel buffer size would be: > >((1 << 18) + ((128 - 1) * (1 << 12))) / 1024 = 764 KB > > This value is ignored when "log_buf_len" kernel parameter is used > as it forces the exact size of the ring buffer to an expected value. > > In order to use num_possible_cpus() we need to make this a late call > on the init process but that also means we cannot share __init code, > the late_setup_log_buf() provided is therefore very similar to the > setup_log_buf() but just modified slightly, in particular since we're > at late boot we can now also use kzalloc(). > > Cc: Michal Hocko > Cc: Petr Mladek > Cc: Andrew Morton > Cc: Joe Perches > Cc: Arun KS > Cc: Kees Cook > Cc: Davidlohr Bueso > Cc: Chris Metcalf > Cc: linux-kernel@vger.kernel.org > Signed-off-by: Luis R. Rodriguez > --- > > This doesn't kfree() at halt but do we need to do it ? > > Documentation/kernel-parameters.txt | 8 -- > include/linux/printk.h | 5 > init/Kconfig| 45 +++- > init/main.c | 1 + > kernel/printk/printk.c | 51 > + > 5 files changed, 107 insertions(+), 3 deletions(-) > > diff --git a/Documentation/kernel-parameters.txt > b/Documentation/kernel-parameters.txt > index 6eaa9cd..229d031 100644 > --- a/Documentation/kernel-parameters.txt > +++ b/Documentation/kernel-parameters.txt > @@ -1685,8 +1685,12 @@ bytes respectively. Such letter suffixes can also be > entirely omitted. > 7 (KERN_DEBUG) debug-level messages > > log_buf_len=n[KMG] Sets the size of the printk ring buffer, > - in bytes. n must be a power of two. The default > - size is set in the kernel config file. > + in bytes. n must be a power of two and greater > + than the minimal size. The minimal size is defined > + by LOG_BUF_SHIFT kernel config parameter. There is > + also CONFIG_LOG_CPU_MIN_BUF_SHIFT config parameter > + that allows to increase the default size depending on > + the number of CPUs. See init/Kconfig for more details. > > logo.nologo [FB] Disables display of the built-in Linux logo. > This may be used to provide more screen space for > diff --git a/include/linux/printk.h b/include/linux/printk.h > index 319ff7e..315a0b7 100644 > --- a/include/linux/printk.h > +++ b/include/linux/printk.h > @@ -161,6 +161,7 @@ extern void wake_up_klogd(void); > > void log_buf_kexec_setup(void); > void __init setup_log_buf(int early); > +void late_setup_log_buf(void); > void
Re: [RFT v4] printk: allow increasing the ring buffer depending on the number of CPUs
On Sat 2014-06-14 11:52:45, Luis R. Rodriguez wrote: From: Luis R. Rodriguez mcg...@suse.com The default size of the ring buffer is too small for machines with a large amount of CPUs under heavy load. What ends up happening when debugging is the ring buffer overlaps and chews up old messages making debugging impossible unless the size is passed as a kernel parameter. An idle system upon boot up will on average spew out only about one or two extra lines but where this really matters is on heavy load and that will vary widely depending on the system and environment. There are mechanisms to help increase the kernel ring buffer for tracing through debugfs, and those interfaces even allow growing the kernel ring buffer per CPU. We also have a static value which can be passed upon boot. Relying on debugfs however is not ideal for production, and relying on the value passed upon bootup is can only used *after* an issue has creeped up. Instead of being reactive this adds a proactive measure which lets you scale the amount of contributions you'd expect to the kernel ring buffer under load by each CPU in the worst case scenario. We use num_possible_cpus() to avoid complexities which could be introduced by dynamically changing the ring buffer size at run time, num_possible_cpus() lets us use the upper limit on possible number of CPUs therefore avoiding having to deal with hotplugging CPUs on and off. This introduces the kernel configuration option LOG_CPU_MIN_BUF_SHIFT which is used to specify the maximum amount of contributions to the kernel ring buffer in the worst case before the kernel ring buffer flips over, the size is specified as a power of 2. The total amount of contributions made by each CPU must be greater than half of the default kernel ring buffer size (1 LOG_BUF_SHIFT bytes) in order to trigger an increase upon bootup. For example if LOG_BUF_SHIFT is 18 (256 KB) you'd require at least 128 KB contributions by other CPUs in order to trigger an increase. With a LOG_CPU_BUF_SHIFT of 12 (4 KB) you'd require at least anything over 64 possible CPUs to trigger an increase. If you had 128 possible CPUs your kernel buffer size would be: ((1 18) + ((128 - 1) * (1 12))) / 1024 = 764 KB This value is ignored when log_buf_len kernel parameter is used as it forces the exact size of the ring buffer to an expected value. In order to use num_possible_cpus() we need to make this a late call on the init process but that also means we cannot share __init code, the late_setup_log_buf() provided is therefore very similar to the setup_log_buf() but just modified slightly, in particular since we're at late boot we can now also use kzalloc(). Cc: Michal Hocko mho...@suse.cz Cc: Petr Mladek pmla...@suse.cz Cc: Andrew Morton a...@linux-foundation.org Cc: Joe Perches j...@perches.com Cc: Arun KS arunks.li...@gmail.com Cc: Kees Cook keesc...@chromium.org Cc: Davidlohr Bueso davidl...@hp.com Cc: Chris Metcalf cmetc...@tilera.com Cc: linux-kernel@vger.kernel.org Signed-off-by: Luis R. Rodriguez mcg...@suse.com --- This doesn't kfree() at halt but do we need to do it ? Documentation/kernel-parameters.txt | 8 -- include/linux/printk.h | 5 init/Kconfig| 45 +++- init/main.c | 1 + kernel/printk/printk.c | 51 + 5 files changed, 107 insertions(+), 3 deletions(-) diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt index 6eaa9cd..229d031 100644 --- a/Documentation/kernel-parameters.txt +++ b/Documentation/kernel-parameters.txt @@ -1685,8 +1685,12 @@ bytes respectively. Such letter suffixes can also be entirely omitted. 7 (KERN_DEBUG) debug-level messages log_buf_len=n[KMG] Sets the size of the printk ring buffer, - in bytes. n must be a power of two. The default - size is set in the kernel config file. + in bytes. n must be a power of two and greater + than the minimal size. The minimal size is defined + by LOG_BUF_SHIFT kernel config parameter. There is + also CONFIG_LOG_CPU_MIN_BUF_SHIFT config parameter + that allows to increase the default size depending on + the number of CPUs. See init/Kconfig for more details. logo.nologo [FB] Disables display of the built-in Linux logo. This may be used to provide more screen space for diff --git a/include/linux/printk.h b/include/linux/printk.h index 319ff7e..315a0b7 100644 --- a/include/linux/printk.h +++ b/include/linux/printk.h @@ -161,6 +161,7 @@ extern void wake_up_klogd(void); void log_buf_kexec_setup(void); void __init setup_log_buf(int early);
Re: [RFT v4] printk: allow increasing the ring buffer depending on the number of CPUs
On Mon, Jun 16, 2014 at 06:32:53PM +0200, Petr Mládek wrote: On Sat 2014-06-14 11:52:45, Luis R. Rodriguez wrote: From: Luis R. Rodriguez mcg...@suse.com The default size of the ring buffer is too small for machines with a large amount of CPUs under heavy load. What ends up happening when debugging is the ring buffer overlaps and chews up old messages making debugging impossible unless the size is passed as a kernel parameter. An idle system upon boot up will on average spew out only about one or two extra lines but where this really matters is on heavy load and that will vary widely depending on the system and environment. There are mechanisms to help increase the kernel ring buffer for tracing through debugfs, and those interfaces even allow growing the kernel ring buffer per CPU. We also have a static value which can be passed upon boot. Relying on debugfs however is not ideal for production, and relying on the value passed upon bootup is can only used *after* an issue has creeped up. Instead of being reactive this adds a proactive measure which lets you scale the amount of contributions you'd expect to the kernel ring buffer under load by each CPU in the worst case scenario. We use num_possible_cpus() to avoid complexities which could be introduced by dynamically changing the ring buffer size at run time, num_possible_cpus() lets us use the upper limit on possible number of CPUs therefore avoiding having to deal with hotplugging CPUs on and off. This introduces the kernel configuration option LOG_CPU_MIN_BUF_SHIFT which is used to specify the maximum amount of contributions to the kernel ring buffer in the worst case before the kernel ring buffer flips over, the size is specified as a power of 2. The total amount of contributions made by each CPU must be greater than half of the default kernel ring buffer size (1 LOG_BUF_SHIFT bytes) in order to trigger an increase upon bootup. For example if LOG_BUF_SHIFT is 18 (256 KB) you'd require at least 128 KB contributions by other CPUs in order to trigger an increase. With a LOG_CPU_BUF_SHIFT of 12 (4 KB) you'd require at least anything over 64 possible CPUs to trigger an increase. If you had 128 possible CPUs your kernel buffer size would be: ((1 18) + ((128 - 1) * (1 12))) / 1024 = 764 KB This value is ignored when log_buf_len kernel parameter is used as it forces the exact size of the ring buffer to an expected value. In order to use num_possible_cpus() we need to make this a late call on the init process but that also means we cannot share __init code, the late_setup_log_buf() provided is therefore very similar to the setup_log_buf() but just modified slightly, in particular since we're at late boot we can now also use kzalloc(). Cc: Michal Hocko mho...@suse.cz Cc: Petr Mladek pmla...@suse.cz Cc: Andrew Morton a...@linux-foundation.org Cc: Joe Perches j...@perches.com Cc: Arun KS arunks.li...@gmail.com Cc: Kees Cook keesc...@chromium.org Cc: Davidlohr Bueso davidl...@hp.com Cc: Chris Metcalf cmetc...@tilera.com Cc: linux-kernel@vger.kernel.org Signed-off-by: Luis R. Rodriguez mcg...@suse.com --- This doesn't kfree() at halt but do we need to do it ? Documentation/kernel-parameters.txt | 8 -- include/linux/printk.h | 5 init/Kconfig| 45 +++- init/main.c | 1 + kernel/printk/printk.c | 51 + 5 files changed, 107 insertions(+), 3 deletions(-) diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt index 6eaa9cd..229d031 100644 --- a/Documentation/kernel-parameters.txt +++ b/Documentation/kernel-parameters.txt @@ -1685,8 +1685,12 @@ bytes respectively. Such letter suffixes can also be entirely omitted. 7 (KERN_DEBUG) debug-level messages log_buf_len=n[KMG] Sets the size of the printk ring buffer, - in bytes. n must be a power of two. The default - size is set in the kernel config file. + in bytes. n must be a power of two and greater + than the minimal size. The minimal size is defined + by LOG_BUF_SHIFT kernel config parameter. There is + also CONFIG_LOG_CPU_MIN_BUF_SHIFT config parameter + that allows to increase the default size depending on + the number of CPUs. See init/Kconfig for more details. logo.nologo [FB] Disables display of the built-in Linux logo. This may be used to provide more screen space for diff --git a/include/linux/printk.h b/include/linux/printk.h index 319ff7e..315a0b7 100644 --- a/include/linux/printk.h +++ b/include/linux/printk.h
