Re: git: 730ea72c706e - main - cam(4): Limit search for disks in SES enclosure by single bus
On Tue, Oct 5, 2021 at 1:55 PM Alexander Motin wrote: > > On 05.10.2021 15:48, Alan Somers wrote: > > On Tue, Oct 5, 2021 at 1:41 PM Alexander Motin wrote: > >> > >> On 05.10.2021 15:14, Alan Somers wrote: > >>> On Tue, Oct 5, 2021 at 1:04 PM Alexander Motin wrote: > > The branch main has been updated by mav: > > URL: > https://cgit.FreeBSD.org/src/commit/?id=730ea72c706ef8e025616772cfd86fd89ed3d42e > > commit 730ea72c706ef8e025616772cfd86fd89ed3d42e > Author: Alexander Motin > AuthorDate: 2021-10-05 19:01:16 + > Commit: Alexander Motin > CommitDate: 2021-10-05 19:01:16 + > > cam(4): Limit search for disks in SES enclosure by single bus > > At least for SAS that we only support now disks are typically > connected to the same bus as the enclosure. Limiting the search > scope makes it much faster on systems with multiple buses and > thousands of disks. > > Reviewed by:imp > MFC after: 2 weeks > Sponsored by: iXsystems, Inc. > Differential Revision: https://reviews.freebsd.org/D32305 > --- > sys/cam/scsi/scsi_enc_ses.c | 8 > 1 file changed, 8 insertions(+) > > diff --git a/sys/cam/scsi/scsi_enc_ses.c b/sys/cam/scsi/scsi_enc_ses.c > >>> > >>> > >>> Sounds juicy. Do you have any benchmarks? > >> > >> I don't think we've benchmarked that system with payload yet, if that is > >> what you mean, polishing usability. But I don't think there can be too > >> much interesting, since big speeds don't require so many HDDs these > >> days, but much smaller number of SSDs instead. This particular system > >> is about capacity -- ~20PiB raw on 1248 disks in one ZFS pool. :) > > > > 1248 disks? That's huge. But I'm not asking about throughput. Your > > commit message said "much faster". What operation is "much faster"? > > Enumerating disks at startup? That would be useful. > > I don't think you notice the difference on startup, too many other > things going on, but during idle it reduces periodic CPU load spikes by > enc_daemon processes by several times, proportionally to the number of > HBAs/buses. More annoyance than a real problem though. > > -- > Alexander Motin Yeah, I've noticed those occasional enc_daemon spikes. I was never sure what it was doing. Thanks for the fix! I'm looking forward to 13.1 a little bit more now.
Re: git: 730ea72c706e - main - cam(4): Limit search for disks in SES enclosure by single bus
On 05.10.2021 15:48, Alan Somers wrote: > On Tue, Oct 5, 2021 at 1:41 PM Alexander Motin wrote: >> >> On 05.10.2021 15:14, Alan Somers wrote: >>> On Tue, Oct 5, 2021 at 1:04 PM Alexander Motin wrote: The branch main has been updated by mav: URL: https://cgit.FreeBSD.org/src/commit/?id=730ea72c706ef8e025616772cfd86fd89ed3d42e commit 730ea72c706ef8e025616772cfd86fd89ed3d42e Author: Alexander Motin AuthorDate: 2021-10-05 19:01:16 + Commit: Alexander Motin CommitDate: 2021-10-05 19:01:16 + cam(4): Limit search for disks in SES enclosure by single bus At least for SAS that we only support now disks are typically connected to the same bus as the enclosure. Limiting the search scope makes it much faster on systems with multiple buses and thousands of disks. Reviewed by:imp MFC after: 2 weeks Sponsored by: iXsystems, Inc. Differential Revision: https://reviews.freebsd.org/D32305 --- sys/cam/scsi/scsi_enc_ses.c | 8 1 file changed, 8 insertions(+) diff --git a/sys/cam/scsi/scsi_enc_ses.c b/sys/cam/scsi/scsi_enc_ses.c >>> >>> >>> Sounds juicy. Do you have any benchmarks? >> >> I don't think we've benchmarked that system with payload yet, if that is >> what you mean, polishing usability. But I don't think there can be too >> much interesting, since big speeds don't require so many HDDs these >> days, but much smaller number of SSDs instead. This particular system >> is about capacity -- ~20PiB raw on 1248 disks in one ZFS pool. :) > > 1248 disks? That's huge. But I'm not asking about throughput. Your > commit message said "much faster". What operation is "much faster"? > Enumerating disks at startup? That would be useful. I don't think you notice the difference on startup, too many other things going on, but during idle it reduces periodic CPU load spikes by enc_daemon processes by several times, proportionally to the number of HBAs/buses. More annoyance than a real problem though. -- Alexander Motin
Re: git: 730ea72c706e - main - cam(4): Limit search for disks in SES enclosure by single bus
On Tue, Oct 5, 2021 at 1:41 PM Alexander Motin wrote: > > On 05.10.2021 15:14, Alan Somers wrote: > > On Tue, Oct 5, 2021 at 1:04 PM Alexander Motin wrote: > >> > >> The branch main has been updated by mav: > >> > >> URL: > >> https://cgit.FreeBSD.org/src/commit/?id=730ea72c706ef8e025616772cfd86fd89ed3d42e > >> > >> commit 730ea72c706ef8e025616772cfd86fd89ed3d42e > >> Author: Alexander Motin > >> AuthorDate: 2021-10-05 19:01:16 + > >> Commit: Alexander Motin > >> CommitDate: 2021-10-05 19:01:16 + > >> > >> cam(4): Limit search for disks in SES enclosure by single bus > >> > >> At least for SAS that we only support now disks are typically > >> connected to the same bus as the enclosure. Limiting the search > >> scope makes it much faster on systems with multiple buses and > >> thousands of disks. > >> > >> Reviewed by:imp > >> MFC after: 2 weeks > >> Sponsored by: iXsystems, Inc. > >> Differential Revision: https://reviews.freebsd.org/D32305 > >> --- > >> sys/cam/scsi/scsi_enc_ses.c | 8 > >> 1 file changed, 8 insertions(+) > >> > >> diff --git a/sys/cam/scsi/scsi_enc_ses.c b/sys/cam/scsi/scsi_enc_ses.c > > > > > > Sounds juicy. Do you have any benchmarks? > > I don't think we've benchmarked that system with payload yet, if that is > what you mean, polishing usability. But I don't think there can be too > much interesting, since big speeds don't require so many HDDs these > days, but much smaller number of SSDs instead. This particular system > is about capacity -- ~20PiB raw on 1248 disks in one ZFS pool. :) > > -- > Alexander Motin 1248 disks? That's huge. But I'm not asking about throughput. Your commit message said "much faster". What operation is "much faster"? Enumerating disks at startup? That would be useful. -Alan
Re: git: 730ea72c706e - main - cam(4): Limit search for disks in SES enclosure by single bus
On 05.10.2021 15:14, Alan Somers wrote: > On Tue, Oct 5, 2021 at 1:04 PM Alexander Motin wrote: >> >> The branch main has been updated by mav: >> >> URL: >> https://cgit.FreeBSD.org/src/commit/?id=730ea72c706ef8e025616772cfd86fd89ed3d42e >> >> commit 730ea72c706ef8e025616772cfd86fd89ed3d42e >> Author: Alexander Motin >> AuthorDate: 2021-10-05 19:01:16 + >> Commit: Alexander Motin >> CommitDate: 2021-10-05 19:01:16 + >> >> cam(4): Limit search for disks in SES enclosure by single bus >> >> At least for SAS that we only support now disks are typically >> connected to the same bus as the enclosure. Limiting the search >> scope makes it much faster on systems with multiple buses and >> thousands of disks. >> >> Reviewed by:imp >> MFC after: 2 weeks >> Sponsored by: iXsystems, Inc. >> Differential Revision: https://reviews.freebsd.org/D32305 >> --- >> sys/cam/scsi/scsi_enc_ses.c | 8 >> 1 file changed, 8 insertions(+) >> >> diff --git a/sys/cam/scsi/scsi_enc_ses.c b/sys/cam/scsi/scsi_enc_ses.c > > > Sounds juicy. Do you have any benchmarks? I don't think we've benchmarked that system with payload yet, if that is what you mean, polishing usability. But I don't think there can be too much interesting, since big speeds don't require so many HDDs these days, but much smaller number of SSDs instead. This particular system is about capacity -- ~20PiB raw on 1248 disks in one ZFS pool. :) -- Alexander Motin
Re: git: 730ea72c706e - main - cam(4): Limit search for disks in SES enclosure by single bus
On Tue, Oct 5, 2021 at 1:04 PM Alexander Motin wrote: > > The branch main has been updated by mav: > > URL: > https://cgit.FreeBSD.org/src/commit/?id=730ea72c706ef8e025616772cfd86fd89ed3d42e > > commit 730ea72c706ef8e025616772cfd86fd89ed3d42e > Author: Alexander Motin > AuthorDate: 2021-10-05 19:01:16 + > Commit: Alexander Motin > CommitDate: 2021-10-05 19:01:16 + > > cam(4): Limit search for disks in SES enclosure by single bus > > At least for SAS that we only support now disks are typically > connected to the same bus as the enclosure. Limiting the search > scope makes it much faster on systems with multiple buses and > thousands of disks. > > Reviewed by:imp > MFC after: 2 weeks > Sponsored by: iXsystems, Inc. > Differential Revision: https://reviews.freebsd.org/D32305 > --- > sys/cam/scsi/scsi_enc_ses.c | 8 > 1 file changed, 8 insertions(+) > > diff --git a/sys/cam/scsi/scsi_enc_ses.c b/sys/cam/scsi/scsi_enc_ses.c Sounds juicy. Do you have any benchmarks?
