Re: [powerpc]Kernel crash while running xfstests (generic/250) [next-20220404]

2022-04-06 Thread Sachin Sant
Comm: dmsetup Not tainted 5.18.0-rc1-next-20220406 #1 [ 2388.287244] NIP: c0154a6c LR: c0154998 CTR: c0674690 [ 2388.287249] REGS: c00145fb3610 TRAP: 0380 Not tainted (5.18.0-rc1-next-20220406) [ 2388.287255] MSR: 8280b033 CR: 28024824 XER: [ 23

Re: [PATCH v2 0/4] Fix perf bench numa, futex and epoll to work with machines having #CPUs > 1K

2022-04-06 Thread Athira Rajeev
> On 07-Apr-2022, at 6:05 AM, Ian Rogers wrote: > > On Wed, Apr 6, 2022 at 10:51 AM Athira Rajeev > wrote: >> >> The perf benchmark for collections: numa, futex and epoll >> hits failure in system configuration with CPU's more than 1024. >> These benchmarks uses "sched_getaffinity" and

Re: [PATCH] testing/selftests/mqueue: Fix mq_perf_tests to free the allocated cpu set

2022-04-06 Thread Athira Rajeev
> On 07-Apr-2022, at 1:35 AM, Shuah Khan wrote: > > On 4/6/22 11:57 AM, Athira Rajeev wrote: >> The selftest "mqueue/mq_perf_tests.c" use CPU_ALLOC to allocate >> CPU set. This cpu set is used further in pthread_attr_setaffinity_np >> and by pthread_create in the code. But in current code,

Re: [PATCH] macintosh: fix via-pmu and via-cuda build errors

2022-04-06 Thread Randy Dunlap
On 4/6/22 19:37, Randy Dunlap wrote: > When CONFIG_INPUT=m, the input_*() family of functions is not > available to builtin drivers. > > When CONFIG_RTC_CLASS is not set, rtc_tm_to_time64() is not defined. > > Fix multiple build errors by making these Kconfig symbols required by > ADB_CUDA

[PATCH] macintosh: fix via-pmu and via-cuda build errors

2022-04-06 Thread Randy Dunlap
When CONFIG_INPUT=m, the input_*() family of functions is not available to builtin drivers. When CONFIG_RTC_CLASS is not set, rtc_tm_to_time64() is not defined. Fix multiple build errors by making these Kconfig symbols required by ADB_CUDA (RTC_CLASS) and ADB_PMU (RTC_CLASS and INPUT).

Re: rcu_sched self-detected stall on CPU

2022-04-06 Thread Zhouyi Zhou
j 8 > > > > 6) qemu-system-ppc64 -kernel vmlinux -nographic -vga none -no-reboot > > > > -smp 2 (QEMU 4.2.1) > > > > 7) after 12 rounds, the bug got reproduced: > > > > (http://154.223.142.244/logs/20220406/qemu.log.txt) > > > > > > J

Re: [PATCH 07/15] x86: remove the IOMMU table infrastructure

2022-04-06 Thread Konrad Rzeszutek Wilk
On Mon, Apr 04, 2022 at 07:05:51AM +0200, Christoph Hellwig wrote: > The IOMMU table tries to separate the different IOMMUs into different > backends, but actually requires various cross calls. > > Rewrite the code to do the generic swiotlb/swiotlb-xen setup directly > in pci-dma.c and then just

Re: [PATCH] ASoC: fsl_sai: fix 1:1 bclk:mclk ratio support

2022-04-06 Thread Shengjiu Wang
On Tue, Apr 5, 2022 at 11:58 PM Ahmad Fatoum wrote: > Refactoring in commit a50b7926d015 ("ASoC: fsl_sai: implement 1:1 > bclk:mclk ratio support") led to the bypass never happening > as (ratio = 1) was caught in the existing if (ratio & 1) continue; > check. The correct check sequence instead

Re: [PATCH 0/4] tools/perf: Fix perf bench numa, futex and epoll to work with machines having #CPUs > 1K

2022-04-06 Thread Disha Goel
-Original Message- From: Athira Rajeev To: a...@kernel.org, jo...@kernel.org, disg...@linux.vnet.ibm.com Cc: m...@ellerman.id.au, linux-perf-us...@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, ma...@linux.vnet.ibm.com, rnsas...@linux.ibm.com, kj...@linux.ibm.com Subject: [PATCH 0/4]

Re: cleanup swiotlb initialization v8

2022-04-06 Thread Konrad Rzeszutek Wilk
On Mon, Apr 04, 2022 at 07:05:44AM +0200, Christoph Hellwig wrote: > Hi all, > > this series tries to clean up the swiotlb initialization, including > that of swiotlb-xen. To get there is also removes the x86 iommu table > infrastructure that massively obsfucates the initialization path. > >

Re: [PATCH 10/15] swiotlb: add a SWIOTLB_ANY flag to lift the low memory restriction

2022-04-06 Thread Konrad Rzeszutek Wilk
> diff --git a/arch/powerpc/platforms/pseries/svm.c > b/arch/powerpc/platforms/pseries/svm.c > index c5228f4969eb2..3b4045d508ec8 100644 > --- a/arch/powerpc/platforms/pseries/svm.c > +++ b/arch/powerpc/platforms/pseries/svm.c > @@ -28,7 +28,7 @@ static int __init init_svm(void) >* need

Re: [PATCH v1 1/1] powerpc/83xx/mpc8349emitx: Get rid of of_node assignment

2022-04-06 Thread Scott Wood
On Wed, 2022-03-23 at 19:43 +0200, Andy Shevchenko wrote: > Let GPIO library to assign of_node from the parent device. > This allows to move GPIO library and drivers to use fwnode > APIs instead of being stuck with OF-only interfaces. > > Signed-off-by: Andy Shevchenko > --- >  

Re: [PATCH] testing/selftests/mqueue: Fix mq_perf_tests to free the allocated cpu set

2022-04-06 Thread Shuah Khan
On 4/6/22 11:57 AM, Athira Rajeev wrote: The selftest "mqueue/mq_perf_tests.c" use CPU_ALLOC to allocate CPU set. This cpu set is used further in pthread_attr_setaffinity_np and by pthread_create in the code. But in current code, allocated cpu set is not freed. Fix this by adding CPU_FREE after

Re: rcu_sched self-detected stall on CPU

2022-04-06 Thread Paul E. McKenney
ar zxf linux-5.18-rc1.tar.gz > > > 3) cp config linux-5.18-rc1/.config > > > 4) cd linux-5.18-rc1 > > > 5) make vmlinux -j 8 > > > 6) qemu-system-ppc64 -kernel vmlinux -nographic -vga none -no-reboot > > > -smp 2 (QEMU 4.2.1) > > > 7) after 12 roun

Re: [PATCH v5 3/8] powerpc/mm/ptdump: debugfs handler for W+X checks at runtime

2022-04-06 Thread Kees Cook
*thread necromancy* Is this patch still something folks are working on? It'd be nice to be able to trigger this check at runtime. -Kees On Wed, Feb 26, 2020 at 05:35:46PM +1100, Russell Currey wrote: > Very rudimentary, just > > echo 1 > [debugfs]/check_wx_pages > > and check the kernel

Re: rcu_sched self-detected stall on CPU

2022-04-06 Thread Zhouyi Zhou
linux -j 8 > > 6) qemu-system-ppc64 -kernel vmlinux -nographic -vga none -no-reboot > > -smp 2 (QEMU 4.2.1) > > 7) after 12 rounds, the bug got reproduced: > > (http://154.223.142.244/logs/20220406/qemu.log.txt) > > Just to make sure, are you both seeing the

[PATCH] testing/selftests/mqueue: Fix mq_perf_tests to free the allocated cpu set

2022-04-06 Thread Athira Rajeev
The selftest "mqueue/mq_perf_tests.c" use CPU_ALLOC to allocate CPU set. This cpu set is used further in pthread_attr_setaffinity_np and by pthread_create in the code. But in current code, allocated cpu set is not freed. Fix this by adding CPU_FREE after its usage is done. Signed-off-by: Athira

[PATCH v2 4/4] tools/perf: Fix perf bench numa testcase to check if CPU used to bind task is online

2022-04-06 Thread Athira Rajeev
Perf numa bench test fails with error: Testcase: ./perf bench numa mem -p 2 -t 1 -P 1024 -C 0,8 -M 1,0 -s 20 -zZq --thp 1 --no-data_rand_walk Failure snippet: <<>> Running 'numa/mem' benchmark: # Running main, "perf bench numa numa-mem -p 2 -t 1 -P 1024 -C 0,8 -M 1,0 -s 20 -zZq --thp 1

[PATCH v2 3/4] tools/perf: Fix perf numa bench to fix usage of affinity for machines with #CPUs > 1K

2022-04-06 Thread Athira Rajeev
perf bench numa testcase fails on systems with CPU's more than 1K. Testcase: perf bench numa mem -p 1 -t 3 -P 512 -s 100 -zZ0qcm --thp 1 Snippet of code: <<>> perf: bench/numa.c:302: bind_to_node: Assertion `!(ret)' failed. Aborted (core dumped) <<>> bind_to_node function uses

[PATCH v2 2/4] tools/perf: Fix perf bench epoll to correct usage of affinity for machines with #CPUs > 1K

2022-04-06 Thread Athira Rajeev
perf bench epoll testcase fails on systems with CPU's more than 1K. Testcase: perf bench epoll all Result snippet: <<>> Run summary [PID 106497]: 1399 threads monitoring on 64 file-descriptors for 8 secs. perf: pthread_create: No such file or directory <<>> In epoll benchmarks (ctl, wait)

[PATCH v2 1/4] tools/perf: Fix perf bench futex to correct usage of affinity for machines with #CPUs > 1K

2022-04-06 Thread Athira Rajeev
perf bench futex testcase fails on systems with CPU's more than 1K. Testcase: perf bench futex all Failure snippet: <<>>Running futex/hash benchmark... perf: pthread_create: No such file or directory <<>> All the futex benchmarks ( ie hash, lock-api, requeue, wake, wake-parallel ),

[PATCH v2 0/4] Fix perf bench numa, futex and epoll to work with machines having #CPUs > 1K

2022-04-06 Thread Athira Rajeev
The perf benchmark for collections: numa, futex and epoll hits failure in system configuration with CPU's more than 1024. These benchmarks uses "sched_getaffinity" and "sched_setaffinity" in the code to work with affinity. Example snippet from numa benchmark: <<>> perf: bench/numa.c:302:

[PATCH] powerpc/perf: Fix the event alternatives array to have correct sort order

2022-04-06 Thread Athira Rajeev
Platform specific PMU supports alternative event for some of the event codes. During perf_event_open, if any event group doesn't match constraint check criteria, further lookup is done to find alternative event. Code checks to see if it is possible to schedule event as group using alternative

Re: rcu_sched self-detected stall on CPU

2022-04-06 Thread Paul E. McKenney
12 rounds, the bug got reproduced: > (http://154.223.142.244/logs/20220406/qemu.log.txt) Just to make sure, are you both seeing the same thing? Last I knew, Zhouyi was chasing an RCU-tasks issue that appears only in kernels built with CONFIG_PROVE_RCU=y, which Miguel does not have set. Or did I mis

Re: [PATCH] powerpc/numa: Handle partially initialized numa nodes

2022-04-06 Thread Oscar Salvador
On Wed, Mar 30, 2022 at 07:21:23PM +0530, Srikar Dronamraju wrote: > arch/powerpc/mm/numa.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c > index b9b7fefbb64b..13022d734951 100644 > --- a/arch/powerpc/mm/numa.c > +++

Re: [PATCH 5/6] powerpc/64: Only WARN if __pa()/__va() called with bad addresses

2022-04-06 Thread Christophe Leroy
Le 06/04/2022 à 16:58, Michael Ellerman a écrit : > We added checks to __pa() / __va() to ensure they're only called with > appropriate addresses. But using BUG_ON() is too strong, it means > virt_addr_valid() will BUG when DEBUG_VIRTUAL is enabled. > > Instead switch them to warnings, arm64

[PATCH 5/6] powerpc/64: Only WARN if __pa()/__va() called with bad addresses

2022-04-06 Thread Michael Ellerman
We added checks to __pa() / __va() to ensure they're only called with appropriate addresses. But using BUG_ON() is too strong, it means virt_addr_valid() will BUG when DEBUG_VIRTUAL is enabled. Instead switch them to warnings, arm64 does the same. Fixes: 4dd7554a6456 ("powerpc/64: Add

[PATCH 2/6] Revert "powerpc: Set max_mapnr correctly"

2022-04-06 Thread Michael Ellerman
From: Kefeng Wang This reverts commit 602946ec2f90d5bd965857753880db29d2d9a1e9. If CONFIG_HIGHMEM is enabled, no highmem will be added with max_mapnr set to max_low_pfn, see mem_init(): for (pfn = highmem_mapnr; pfn < max_mapnr; ++pfn) { ... free_highmem_page(); } Now that

[RFC PATCH 6/6] powerpc/mm: Add virt_addr_valid() checks

2022-04-06 Thread Michael Ellerman
We've had several bugs now with virt_addr_valid() being wrong, so lets add some always-enabled boot time checks that it behaves as expected. Signed-off-by: Michael Ellerman --- arch/powerpc/mm/mem.c | 7 +++ 1 file changed, 7 insertions(+) diff --git a/arch/powerpc/mm/mem.c

[PATCH 4/6] powerpc/vas: Fix __pa() handling in init_winctx_regs()

2022-04-06 Thread Michael Ellerman
In init_winctx_regs() we call __pa() on winctx->rx_fifo, but some callers pass a real adress which causes errors with DEBUG_VIRTUAL enabled. So check first if we have a virtual address, and otherwise leave the address unchanged. Signed-off-by: Michael Ellerman ---

[PATCH 3/6] powerpc/85xx: Fix virt_to_phys() off-by-one in smp_85xx_start_cpu()

2022-04-06 Thread Michael Ellerman
In smp_85xx_start_cpu() we are passed an address but we're unsure if it's a real or virtual address, so there's a check to determine that. The check has an off-by-one in that it tests if the address is greater than high_memory, but high_memory is the first address of high memory, so the check

[PATCH 1/6] powerpc: Fix virt_addr_valid() for 64-bit Book3E & 32-bit

2022-04-06 Thread Michael Ellerman
From: Kefeng Wang mpe: On 64-bit Book3E vmalloc space starts at 0x8000. Because of the way __pa() works we have: __pa(0x8000) == 0, and therefore virt_to_pfn(0x8000) == 0, and therefore virt_addr_valid(0x8000) == true Which is wrong,

Re: [PATCH v1 1/1] powerpc/83xx/mpc8349emitx: Get rid of of_node assignment

2022-04-06 Thread Andy Shevchenko
On Mon, Mar 28, 2022 at 03:16:08PM +0200, Linus Walleij wrote: > On Wed, Mar 23, 2022 at 6:43 PM Andy Shevchenko > wrote: > > > Let GPIO library to assign of_node from the parent device. > > This allows to move GPIO library and drivers to use fwnode > > APIs instead of being stuck with OF-only

Re: [PATCH kernel] KVM: PPC: Fix TCE handling for VFIO

2022-04-06 Thread Frederic Barrat
On 06/04/2022 06:04, Alexey Kardashevskiy wrote: At the moment the IOMMU page size in a pseries VM is 16MB (the biggest allowed by LoPAPR), this page size is used for an emulated TCE table. If there is a passed though PCI device, that there are hardware IOMMU tables with equal or smaller

Re: [PATCH net-next] orinoco: Prepare cleanup of powerpc's asm/prom.h

2022-04-06 Thread Kalle Valo
Christophe Leroy wrote: > powerpc's asm/prom.h brings some headers that it doesn't > need itself. > > In order to clean it up, first add missing headers in > users of asm/prom.h > > Signed-off-by: Christophe Leroy Patch applied to wireless-next.git, thanks. 3223e922ccf8 orinoco: Prepare

[PATCH 3/3] powerpc/fadump: print start of preserved area

2022-04-06 Thread Hari Bathini
Print preserved area start address in fadump_region_show() function. Signed-off-by: Hari Bathini --- arch/powerpc/platforms/powernv/opal-fadump.c | 6 +++--- arch/powerpc/platforms/pseries/rtas-fadump.c | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git

[PATCH 2/3] powerpc/fadump: align destination address to pagesize

2022-04-06 Thread Hari Bathini
On crash, boot memory area is copied to a destination address by f/w. This region is setup as separate PT_LOAD segment with appropriate offset to handle the different physical address and offset in vmcore. If this destination address is not page aligned, reading the vmcore with mmap is likely to

[PATCH 1/3] powerpc/fadump: fix PT_LOAD segment for boot memory area

2022-04-06 Thread Hari Bathini
Boot memory area is setup as separate PT_LOAD segment in the vmcore as it is moved by f/w, on crash, to a destination address provided by the kernel. Having separate PT_LOAD segment helps in handling the different physical address and offset for boot memory area in the vmcore. Commit ced1bf52f477

[PATCH 0/3] powerpc/fadump: fadump fixes and improvements

2022-04-06 Thread Hari Bathini
This series fixes a couple of issues in fadump and improves logging. The first patch fixes a bug that is observed when memory reserved for fadump is not adjacent to boot memory area. The second patch ensures mmap read succeeds for all memory regions in the vmcore. The third patch improves the data

Re: rcu_sched self-detected stall on CPU

2022-04-06 Thread Zhouyi Zhou
linux-5.18-rc1/.config 4) cd linux-5.18-rc1 5) make vmlinux -j 8 6) qemu-system-ppc64 -kernel vmlinux -nographic -vga none -no-reboot -smp 2 (QEMU 4.2.1) 7) after 12 rounds, the bug got reproduced: (http://154.223.142.244/logs/20220406/qemu.log.txt) Cheers ;-) Zhouyi On Wed, Apr 6, 2022 at 3:47 PM

Re: [PATCH] cxl/ocxl: Prepare cleanup of powerpc's asm/prom.h

2022-04-06 Thread Frederic Barrat
On 02/04/2022 11:52, Christophe Leroy wrote: powerpc's asm/prom.h brings some headers that it doesn't need itself. In order to clean it up, first add missing headers in users of asm/prom.h Signed-off-by: Christophe Leroy --- LGTM Acked-by: Frederic Barrat drivers/misc/cxl/api.c

Re: [PATCH] tty: serial: Prepare cleanup of powerpc's asm/prom.h

2022-04-06 Thread Jiri Slaby
On 02. 04. 22, 12:20, Christophe Leroy wrote: powerpc's asm/prom.h brings some headers that it doesn't need itself. In order to clean it up, first add missing headers in users of asm/prom.h Signed-off-by: Christophe Leroy Reviewed-by: Jiri Slaby ---

Re: [PATCH 2/4] tools/perf: Fix perf bench epoll to correct usage of affinity for machines with #CPUs > 1K

2022-04-06 Thread Athira Rajeev
> On 05-Apr-2022, at 11:26 PM, Ian Rogers wrote: > > On Fri, Apr 1, 2022 at 12:00 PM Athira Rajeev > wrote: >> >> perf bench epoll testcase fails on systems with CPU's >> more than 1K. >> >> Testcase: perf bench epoll all >> Result snippet: >> <<>> >> Run summary [PID 106497]: 1399 threads

Re: [PATCH 3/4] tools/perf: Fix perf numa bench to fix usage of affinity for machines with #CPUs > 1K

2022-04-06 Thread Athira Rajeev
> On 05-Apr-2022, at 11:22 PM, Ian Rogers wrote: > > On Fri, Apr 1, 2022 at 11:59 AM Athira Rajeev > wrote: >> >> perf bench numa testcase fails on systems with CPU's >> more than 1K. >> >> Testcase: perf bench numa mem -p 1 -t 3 -P 512 -s 100 -zZ0qcm --thp 1 >> Snippet of code: >> <<>>

Re: [PATCH] powerpc/boot: Build wrapper for an appropriate CPU

2022-04-06 Thread Joel Stanley
On Thu, 31 Mar 2022 at 23:46, Segher Boessenkool wrote: > > On Thu, Mar 31, 2022 at 12:19:52PM -0300, Murilo Opsfelder Araújo wrote: > > My understanding is that the default cpu type for -mcpu=powerpc64 can > > change. > > Different subtargets (Linux, AIX, Darwin, the various BSDs, bare ELF, >

[PATCH kernel v3] powerpc/boot: Stop using RELACOUNT

2022-04-06 Thread Alexey Kardashevskiy
So far the RELACOUNT tag from the ELF header was containing the exact number of R_PPC_RELATIVE/R_PPC64_RELATIVE relocations. However the LLVM's recent change [1] make it equal-or-less than the actual number which makes it useless. This replaces RELACOUNT in zImage loader with a pair of RELASZ and

Re: [PATCH kernel v2] powerpc/boot: Stop using RELACOUNT

2022-04-06 Thread Alexey Kardashevskiy
On 4/6/22 14:58, Gabriel Paubert wrote: On Wed, Apr 06, 2022 at 02:01:48PM +1000, Alexey Kardashevskiy wrote: So far the RELACOUNT tag from the ELF header was containing the exact number of R_PPC_RELATIVE/R_PPC64_RELATIVE relocations. However the LLVM's recent change [1] make it

Re: [PATCH] cxl/ocxl: Prepare cleanup of powerpc's asm/prom.h

2022-04-06 Thread Andrew Donnellan
On 6/4/22 16:52, Christophe Leroy wrote: Thanks. Can you take the patch through your tree ? We don't have a tree of our own, we rely on mpe to take it through powerpc. -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited

Re: [PATCH] cxl/ocxl: Prepare cleanup of powerpc's asm/prom.h

2022-04-06 Thread Christophe Leroy
Le 06/04/2022 à 08:43, Andrew Donnellan a écrit : > On 2/4/22 20:52, Christophe Leroy wrote: >> powerpc's asm/prom.h brings some headers that it doesn't >> need itself. >> >> In order to clean it up, first add missing headers in >> users of asm/prom.h >> >> Signed-off-by: Christophe Leroy > >

Re: [PATCH] cxl/ocxl: Prepare cleanup of powerpc's asm/prom.h

2022-04-06 Thread Andrew Donnellan
On 2/4/22 20:52, Christophe Leroy wrote: powerpc's asm/prom.h brings some headers that it doesn't need itself. In order to clean it up, first add missing headers in users of asm/prom.h Signed-off-by: Christophe Leroy Untested because I don't have your actual patch to change prom.h, but

[PATCH v2 5/5] powerpc/8xx: Use kmalloced data structure instead of global static

2022-04-06 Thread Christophe Leroy
Use a kmalloced data structure to store interrupt controller internal data instead of static global variables. Signed-off-by: Christophe Leroy --- arch/powerpc/platforms/8xx/cpm1-ic.c | 48 +--- 1 file changed, 30 insertions(+), 18 deletions(-) diff --git

[PATCH v2 3/5] powerpc/8xx: Convert CPM1 interrupt controller to platform_device

2022-04-06 Thread Christophe Leroy
In the same logic as commit be7ecbd240b2 ("soc: fsl: qe: convert QE interrupt controller to platform_device"), convert CPM1 interrupt controller to platform_device. Signed-off-by: Christophe Leroy --- arch/powerpc/platforms/8xx/cpm1-ic.c| 90 ++---

[PATCH v2 4/5] powerpc/8xx: Remove mpc8xx_pics_init()

2022-04-06 Thread Christophe Leroy
mpc8xx_pics_init() is now only a trampoline to mpc8xx_pic_init(). Remove mpc8xx_pics_init() and use mpc8xx_pic_init() directly. Signed-off-by: Christophe Leroy --- v2: Add missing header (pic.h) and change mpc8xx_pic_init() to void. --- arch/powerpc/platforms/8xx/adder875.c| 3 ++-

[PATCH v2 2/5] powerpc/8xx: Convert CPM1 error interrupt handler to platform driver

2022-04-06 Thread Christophe Leroy
Add CPM error interrupt as a standalone platform driver, to simplify the init of CPM interrupt handler. Signed-off-by: Christophe Leroy --- arch/powerpc/platforms/8xx/cpm1-ic.c | 73 +--- 1 file changed, 44 insertions(+), 29 deletions(-) diff --git

[PATCH v2 1/5] powerpc/8xx: Move CPM interrupt controller into a dedicated file

2022-04-06 Thread Christophe Leroy
CPM interrupt controller is quite standalone. Move it into a dedicated file. It will help for next step which will change it to a platform driver. This is pure code move, checkpatch report is ignored at this point, except one parenthesis alignment which would remain at the end of the series. All