Re: ndctl tests usable?
On Thu, Apr 28, 2022 at 10:02:47AM +0530, Shivaprasad G Bhat wrote: > On 4/26/22 23:39, Michal Suchánek wrote: > > On Tue, Apr 26, 2022 at 09:47:19AM -0700, Dan Williams wrote: > > > On Tue, Apr 26, 2022 at 9:43 AM Michal Suchánek wrote: > > > > > > > > On Tue, Apr 26, 2022 at 09:32:24AM -0700, Dan Williams wrote: > > > > > On Tue, Apr 26, 2022 at 9:15 AM Michal Suchánek > > > > > wrote: > > > > > > > > > > > > On Tue, Apr 26, 2022 at 08:51:25AM -0700, Dan Williams wrote: > > > > > > > On Tue, Apr 26, 2022 at 5:39 AM Michal Suchánek > > > > > > > wrote: > > > > > > > > > > ... > > > > > > > > > > The modinfo just tells you what modules are available, but it does not > > > > > necessarily indicate which modules are actively loaded in the system > > > > > which is what ndctl_test_init() validates. > > > > > > > > Isn't what modinfo lists also what modrobe loads? > > > > > modules are not loaded before the test. > > > > Maybe something goes wrong with the test module build? > > > > It is very fragile and requires complete kernel source for each > > configuration built. See below for the package > > > > https://build.opensuse.org/package/show/home:michals/nfit_test > > > > Attaching the log of test run which does not report any missing tools, > > only complains about nfit.ko being production. > > I have attempted to fix few of the known issues in the first 3 patches of > the series posted here. > > https://patchwork.kernel.org/project/linux-nvdimm/list/?series=633103 Thanks for pointing out these series. It is certainly of interest but given the test VM is x86 the papr fixes don't really apply to this case. Michal
Re: ndctl tests usable?
On 4/26/22 23:39, Michal Suchánek wrote: On Tue, Apr 26, 2022 at 09:47:19AM -0700, Dan Williams wrote: On Tue, Apr 26, 2022 at 9:43 AM Michal Suchánek wrote: On Tue, Apr 26, 2022 at 09:32:24AM -0700, Dan Williams wrote: On Tue, Apr 26, 2022 at 9:15 AM Michal Suchánek wrote: On Tue, Apr 26, 2022 at 08:51:25AM -0700, Dan Williams wrote: On Tue, Apr 26, 2022 at 5:39 AM Michal Suchánek wrote: ... The modinfo just tells you what modules are available, but it does not necessarily indicate which modules are actively loaded in the system which is what ndctl_test_init() validates. Isn't what modinfo lists also what modrobe loads? modules are not loaded before the test. Maybe something goes wrong with the test module build? It is very fragile and requires complete kernel source for each configuration built. See below for the package https://build.opensuse.org/package/show/home:michals/nfit_test Attaching the log of test run which does not report any missing tools, only complains about nfit.ko being production. I have attempted to fix few of the known issues in the first 3 patches of the series posted here. https://patchwork.kernel.org/project/linux-nvdimm/list/?series=633103 Thanks Michal
Re: ndctl tests usable?
On Tue, Apr 26, 2022 at 11:22:55AM -0700, Dan Williams wrote:
> On Tue, Apr 26, 2022 at 11:10 AM Michal Suchánek wrote:
> >
> > On Tue, Apr 26, 2022 at 09:47:19AM -0700, Dan Williams wrote:
> > > On Tue, Apr 26, 2022 at 9:43 AM Michal Suchánek wrote:
> > > >
> > > > On Tue, Apr 26, 2022 at 09:32:24AM -0700, Dan Williams wrote:
> > > > > On Tue, Apr 26, 2022 at 9:15 AM Michal Suchánek
> > > > > wrote:
> > > > > >
> > > > > > On Tue, Apr 26, 2022 at 08:51:25AM -0700, Dan Williams wrote:
> > > > > > > On Tue, Apr 26, 2022 at 5:39 AM Michal Suchánek
> > > > > > > wrote:
> > > > > > > >
> > ...
> > > > >
> > > > > The modinfo just tells you what modules are available, but it does not
> > > > > necessarily indicate which modules are actively loaded in the system
> > > > > which is what ndctl_test_init() validates.
> > > >
> > > > Isn't what modinfo lists also what modrobe loads?
> > >
> > > It shows what modprobe would load on the next invocation, but
> > > sometimes when nfit_test fails it's because the initramfs or something
> > > else loaded the modules without respecting the extra/ (or updates/ in
> > > your case) override modules.
> > >
> > > > There isn't any pmem so I don't see why production modules would be
> > > > loaded before the test modules are installed. Unloading the modules
> > > > first does not really make any difference.
> > >
> > > Ok, my first guess was wrong... would need more debug to see what
> > > those other skip tests are complaining about.
> >
> > There was also missing parted and hostname command.
> >
> > However, the nfit.ko is detected as production even when I remove all
> > the production modules just in case. lsmod confirms that the nvdimm
> > modules are not loaded before the test.
> >
> > Maybe something goes wrong with the test module build?
> >
> > It is very fragile and requires complete kernel source for each
> > configuration built. See below for the package
> >
> > https://build.opensuse.org/package/show/home:michals/nfit_test
> >
> > Attaching the log of test run which does not report any missing tools,
> > only complains about nfit.ko being production.
>
> Oh... something silly, ndctl_test_init() assumes that the out-of-tree
> module directory is always "/lib/modules/$(uname -r)/extra"
>
> if (!strstr(path, "/extra/")) {
> log_err(_ctx, "%s.ko: appears to be
> production version: %s\n",
> name, path);
> break;
> }
>
> Looks like a build configuration variable is needed there to allow for
> "updates/".
yes, I could also change the path in the package to match this
expactation. Not sure if that would break something else.
There is one last bit: the tests require at least 1GB of ram, or
specifically the default VM with 512MB is not sufficient.
With that only monitor test fails, and some BTT test is skipped because
the 5.17 kernel supposedly does not have the correct support:
[ 182s] 19/23 ndctl:ndctl / monitor.sh FAIL 6.39s exit status 1
[ 182s]
[ 182s]
[ 182s] Ok: 21
[ 182s] Expected Fail: 0
[ 182s] Fail: 1
[ 182s] Unexpected Pass:0
[ 182s] Skipped:1
[ 182s] Timeout:0
Thanks
Michal
Re: ndctl tests usable?
On Tue, Apr 26, 2022 at 11:10 AM Michal Suchánek wrote:
>
> On Tue, Apr 26, 2022 at 09:47:19AM -0700, Dan Williams wrote:
> > On Tue, Apr 26, 2022 at 9:43 AM Michal Suchánek wrote:
> > >
> > > On Tue, Apr 26, 2022 at 09:32:24AM -0700, Dan Williams wrote:
> > > > On Tue, Apr 26, 2022 at 9:15 AM Michal Suchánek
> > > > wrote:
> > > > >
> > > > > On Tue, Apr 26, 2022 at 08:51:25AM -0700, Dan Williams wrote:
> > > > > > On Tue, Apr 26, 2022 at 5:39 AM Michal Suchánek
> > > > > > wrote:
> > > > > > >
> ...
> > > >
> > > > The modinfo just tells you what modules are available, but it does not
> > > > necessarily indicate which modules are actively loaded in the system
> > > > which is what ndctl_test_init() validates.
> > >
> > > Isn't what modinfo lists also what modrobe loads?
> >
> > It shows what modprobe would load on the next invocation, but
> > sometimes when nfit_test fails it's because the initramfs or something
> > else loaded the modules without respecting the extra/ (or updates/ in
> > your case) override modules.
> >
> > > There isn't any pmem so I don't see why production modules would be
> > > loaded before the test modules are installed. Unloading the modules
> > > first does not really make any difference.
> >
> > Ok, my first guess was wrong... would need more debug to see what
> > those other skip tests are complaining about.
>
> There was also missing parted and hostname command.
>
> However, the nfit.ko is detected as production even when I remove all
> the production modules just in case. lsmod confirms that the nvdimm
> modules are not loaded before the test.
>
> Maybe something goes wrong with the test module build?
>
> It is very fragile and requires complete kernel source for each
> configuration built. See below for the package
>
> https://build.opensuse.org/package/show/home:michals/nfit_test
>
> Attaching the log of test run which does not report any missing tools,
> only complains about nfit.ko being production.
Oh... something silly, ndctl_test_init() assumes that the out-of-tree
module directory is always "/lib/modules/$(uname -r)/extra"
if (!strstr(path, "/extra/")) {
log_err(_ctx, "%s.ko: appears to be
production version: %s\n",
name, path);
break;
}
Looks like a build configuration variable is needed there to allow for
"updates/".
Re: ndctl tests usable?
On Tue, Apr 26, 2022 at 09:47:19AM -0700, Dan Williams wrote: > On Tue, Apr 26, 2022 at 9:43 AM Michal Suchánek wrote: > > > > On Tue, Apr 26, 2022 at 09:32:24AM -0700, Dan Williams wrote: > > > On Tue, Apr 26, 2022 at 9:15 AM Michal Suchánek wrote: > > > > > > > > On Tue, Apr 26, 2022 at 08:51:25AM -0700, Dan Williams wrote: > > > > > On Tue, Apr 26, 2022 at 5:39 AM Michal Suchánek > > > > > wrote: > > > > > > ... > > > > > > The modinfo just tells you what modules are available, but it does not > > > necessarily indicate which modules are actively loaded in the system > > > which is what ndctl_test_init() validates. > > > > Isn't what modinfo lists also what modrobe loads? > > It shows what modprobe would load on the next invocation, but > sometimes when nfit_test fails it's because the initramfs or something > else loaded the modules without respecting the extra/ (or updates/ in > your case) override modules. > > > There isn't any pmem so I don't see why production modules would be > > loaded before the test modules are installed. Unloading the modules > > first does not really make any difference. > > Ok, my first guess was wrong... would need more debug to see what > those other skip tests are complaining about. There was also missing parted and hostname command. However, the nfit.ko is detected as production even when I remove all the production modules just in case. lsmod confirms that the nvdimm modules are not loaded before the test. Maybe something goes wrong with the test module build? It is very fragile and requires complete kernel source for each configuration built. See below for the package https://build.opensuse.org/package/show/home:michals/nfit_test Attaching the log of test run which does not report any missing tools, only complains about nfit.ko being production. Thanks Michal ndctl-test.log.xz Description: application/xz
Re: ndctl tests usable?
On Tue, Apr 26, 2022 at 9:43 AM Michal Suchánek wrote: > > On Tue, Apr 26, 2022 at 09:32:24AM -0700, Dan Williams wrote: > > On Tue, Apr 26, 2022 at 9:15 AM Michal Suchánek wrote: > > > > > > On Tue, Apr 26, 2022 at 08:51:25AM -0700, Dan Williams wrote: > > > > On Tue, Apr 26, 2022 at 5:39 AM Michal Suchánek > > > > wrote: > > > > > > > > > > Hello, > > > > > > > > > > there is some testsuite included with ndctl, and when following the > > > > > instructions to build it most tests fail or are skipped: > > > > > > > > > > [ 95s] Ok: 3 > > > > > [ 95s] Expected Fail: 0 > > > > > [ 95s] Fail: 5 > > > > > [ 95s] Unexpected Pass:0 > > > > > [ 95s] Skipped:15 > > > > > [ 95s] Timeout:0 > > > > > > > > > > Is this the expected outcome or is this a problem with the ndctl > > > > > build? > > > > > > > > > > Attaching test run log. > > > > > > > > I see a few missing prerequisites: > > > > > > > > [ 78s] /usr/src/packages/BUILD/ndctl-73/test/pmem-errors.sh: line > > > > 64: mkfs.ext4: command not found > > > > [ 95s] /usr/src/packages/BUILD/ndctl-73/test/security.sh: line 25: > > > > jq: command not found > > > > > > Indeed, with those installed I get much more tests passing: > > > > > > [ 148s] Ok: 13 > > > [ 148s] Expected Fail: 0 > > > [ 148s] Fail: 4 > > > [ 148s] Unexpected Pass:0 > > > [ 148s] Skipped:6 > > > [ 148s] Timeout:0 > > > > > > > > > > > This report: > > > > > > > > [ 51s] 1/23 ndctl:ndctl / libndctl SKIP > > > > 0.02s exit status 77 > > > > > > > > ...seems to indicate that the nfit_test modules did not appear to load > > > > correctly. I never expected that the nfit_test modules would be > > > > redistributable, so I was surprised to see them being installed by an > > > > actual package "nfit_test-kmp-default-0_k5.17.4_1-6.1". The reason > > > > they are not redistributable is because they require replacing the > > > > production build of the kernel provided modules libnvdimm.ko, > > > > nd_pmem.ko, etc... What I expect is happening is that the production > > > > version of libnvdimm.ko is already loaded (or is the only one on the > > > > > > AFAICT neither is the case, that's why I dump the module information in > > > the log. > > > > The modinfo just tells you what modules are available, but it does not > > necessarily indicate which modules are actively loaded in the system > > which is what ndctl_test_init() validates. > > Isn't what modinfo lists also what modrobe loads? It shows what modprobe would load on the next invocation, but sometimes when nfit_test fails it's because the initramfs or something else loaded the modules without respecting the extra/ (or updates/ in your case) override modules. > There isn't any pmem so I don't see why production modules would be > loaded before the test modules are installed. Unloading the modules > first does not really make any difference. Ok, my first guess was wrong... would need more debug to see what those other skip tests are complaining about.
Re: ndctl tests usable?
On Tue, Apr 26, 2022 at 09:32:24AM -0700, Dan Williams wrote: > On Tue, Apr 26, 2022 at 9:15 AM Michal Suchánek wrote: > > > > On Tue, Apr 26, 2022 at 08:51:25AM -0700, Dan Williams wrote: > > > On Tue, Apr 26, 2022 at 5:39 AM Michal Suchánek wrote: > > > > > > > > Hello, > > > > > > > > there is some testsuite included with ndctl, and when following the > > > > instructions to build it most tests fail or are skipped: > > > > > > > > [ 95s] Ok: 3 > > > > [ 95s] Expected Fail: 0 > > > > [ 95s] Fail: 5 > > > > [ 95s] Unexpected Pass:0 > > > > [ 95s] Skipped:15 > > > > [ 95s] Timeout:0 > > > > > > > > Is this the expected outcome or is this a problem with the ndctl build? > > > > > > > > Attaching test run log. > > > > > > I see a few missing prerequisites: > > > > > > [ 78s] /usr/src/packages/BUILD/ndctl-73/test/pmem-errors.sh: line > > > 64: mkfs.ext4: command not found > > > [ 95s] /usr/src/packages/BUILD/ndctl-73/test/security.sh: line 25: > > > jq: command not found > > > > Indeed, with those installed I get much more tests passing: > > > > [ 148s] Ok: 13 > > [ 148s] Expected Fail: 0 > > [ 148s] Fail: 4 > > [ 148s] Unexpected Pass:0 > > [ 148s] Skipped:6 > > [ 148s] Timeout:0 > > > > > > > > This report: > > > > > > [ 51s] 1/23 ndctl:ndctl / libndctl SKIP > > > 0.02s exit status 77 > > > > > > ...seems to indicate that the nfit_test modules did not appear to load > > > correctly. I never expected that the nfit_test modules would be > > > redistributable, so I was surprised to see them being installed by an > > > actual package "nfit_test-kmp-default-0_k5.17.4_1-6.1". The reason > > > they are not redistributable is because they require replacing the > > > production build of the kernel provided modules libnvdimm.ko, > > > nd_pmem.ko, etc... What I expect is happening is that the production > > > version of libnvdimm.ko is already loaded (or is the only one on the > > > > AFAICT neither is the case, that's why I dump the module information in > > the log. > > The modinfo just tells you what modules are available, but it does not > necessarily indicate which modules are actively loaded in the system > which is what ndctl_test_init() validates. Isn't what modinfo lists also what modrobe loads? There isn't any pmem so I don't see why production modules would be loaded before the test modules are installed. Unloading the modules first does not really make any difference. Thanks Michal
Re: ndctl tests usable?
On Tue, Apr 26, 2022 at 9:15 AM Michal Suchánek wrote: > > On Tue, Apr 26, 2022 at 08:51:25AM -0700, Dan Williams wrote: > > On Tue, Apr 26, 2022 at 5:39 AM Michal Suchánek wrote: > > > > > > Hello, > > > > > > there is some testsuite included with ndctl, and when following the > > > instructions to build it most tests fail or are skipped: > > > > > > [ 95s] Ok: 3 > > > [ 95s] Expected Fail: 0 > > > [ 95s] Fail: 5 > > > [ 95s] Unexpected Pass:0 > > > [ 95s] Skipped:15 > > > [ 95s] Timeout:0 > > > > > > Is this the expected outcome or is this a problem with the ndctl build? > > > > > > Attaching test run log. > > > > I see a few missing prerequisites: > > > > [ 78s] /usr/src/packages/BUILD/ndctl-73/test/pmem-errors.sh: line > > 64: mkfs.ext4: command not found > > [ 95s] /usr/src/packages/BUILD/ndctl-73/test/security.sh: line 25: > > jq: command not found > > Indeed, with those installed I get much more tests passing: > > [ 148s] Ok: 13 > [ 148s] Expected Fail: 0 > [ 148s] Fail: 4 > [ 148s] Unexpected Pass:0 > [ 148s] Skipped:6 > [ 148s] Timeout:0 > > > > > This report: > > > > [ 51s] 1/23 ndctl:ndctl / libndctl SKIP > > 0.02s exit status 77 > > > > ...seems to indicate that the nfit_test modules did not appear to load > > correctly. I never expected that the nfit_test modules would be > > redistributable, so I was surprised to see them being installed by an > > actual package "nfit_test-kmp-default-0_k5.17.4_1-6.1". The reason > > they are not redistributable is because they require replacing the > > production build of the kernel provided modules libnvdimm.ko, > > nd_pmem.ko, etc... What I expect is happening is that the production > > version of libnvdimm.ko is already loaded (or is the only one on the > > AFAICT neither is the case, that's why I dump the module information in > the log. The modinfo just tells you what modules are available, but it does not necessarily indicate which modules are actively loaded in the system which is what ndctl_test_init() validates.
Re: ndctl tests usable?
On Tue, Apr 26, 2022 at 08:51:25AM -0700, Dan Williams wrote: > On Tue, Apr 26, 2022 at 5:39 AM Michal Suchánek wrote: > > > > Hello, > > > > there is some testsuite included with ndctl, and when following the > > instructions to build it most tests fail or are skipped: > > > > [ 95s] Ok: 3 > > [ 95s] Expected Fail: 0 > > [ 95s] Fail: 5 > > [ 95s] Unexpected Pass:0 > > [ 95s] Skipped:15 > > [ 95s] Timeout:0 > > > > Is this the expected outcome or is this a problem with the ndctl build? > > > > Attaching test run log. > > I see a few missing prerequisites: > > [ 78s] /usr/src/packages/BUILD/ndctl-73/test/pmem-errors.sh: line > 64: mkfs.ext4: command not found > [ 95s] /usr/src/packages/BUILD/ndctl-73/test/security.sh: line 25: > jq: command not found Indeed, with those installed I get much more tests passing: [ 148s] Ok: 13 [ 148s] Expected Fail: 0 [ 148s] Fail: 4 [ 148s] Unexpected Pass:0 [ 148s] Skipped:6 [ 148s] Timeout:0 > > This report: > > [ 51s] 1/23 ndctl:ndctl / libndctl SKIP > 0.02s exit status 77 > > ...seems to indicate that the nfit_test modules did not appear to load > correctly. I never expected that the nfit_test modules would be > redistributable, so I was surprised to see them being installed by an > actual package "nfit_test-kmp-default-0_k5.17.4_1-6.1". The reason > they are not redistributable is because they require replacing the > production build of the kernel provided modules libnvdimm.ko, > nd_pmem.ko, etc... What I expect is happening is that the production > version of libnvdimm.ko is already loaded (or is the only one on the AFAICT neither is the case, that's why I dump the module information in the log. Thanks Michal
Re: ndctl tests usable?
On Tue, Apr 26, 2022 at 5:39 AM Michal Suchánek wrote: > > Hello, > > there is some testsuite included with ndctl, and when following the > instructions to build it most tests fail or are skipped: > > [ 95s] Ok: 3 > [ 95s] Expected Fail: 0 > [ 95s] Fail: 5 > [ 95s] Unexpected Pass:0 > [ 95s] Skipped:15 > [ 95s] Timeout:0 > > Is this the expected outcome or is this a problem with the ndctl build? > > Attaching test run log. I see a few missing prerequisites: [ 78s] /usr/src/packages/BUILD/ndctl-73/test/pmem-errors.sh: line 64: mkfs.ext4: command not found [ 95s] /usr/src/packages/BUILD/ndctl-73/test/security.sh: line 25: jq: command not found This report: [ 51s] 1/23 ndctl:ndctl / libndctl SKIP 0.02s exit status 77 ...seems to indicate that the nfit_test modules did not appear to load correctly. I never expected that the nfit_test modules would be redistributable, so I was surprised to see them being installed by an actual package "nfit_test-kmp-default-0_k5.17.4_1-6.1". The reason they are not redistributable is because they require replacing the production build of the kernel provided modules libnvdimm.ko, nd_pmem.ko, etc... What I expect is happening is that the production version of libnvdimm.ko is already loaded (or is the only one on the system), and ndctl_test_init() (https://github.com/pmem/ndctl/blob/main/test/core.c#L110) detects that case and skips the tests. This is what I see with my setup that uses the sequence below to install nfit_test and friends: Test summary: Ok: 35 Expected Fail: 0 Fail: 1 Unexpected Pass:0 Skipped:1 Timeout:0 Install procedure from a kernel build directory: export INSTALL_MOD_PATH=$root make INSTALL_HDR_PATH=$root/usr headers_install make M=tools/testing/nvdimm modules_install make M=tools/testing/cxl modules_install make modules_install
