Re: [PATCH v2 2/2] drm/tests: Split drm_test_dp_mst_sideband_msg_req_decode into parameterized tests
On 9/30/22 06:11, Michał Winiarski wrote: > On Fri, Sep 30, 2022 at 02:50:43PM +0800, David Gow wrote: >> On Fri, Sep 30, 2022 at 6:33 AM Michał Winiarski >> wrote: >>> >>> On Tue, Sep 27, 2022 at 07:12:06PM -0300, Maíra Canal wrote: The drm_test_dp_mst_sideband_msg_req_decode repeats the same test structure with different parameters. This could be better represented by parameterized tests, provided by KUnit. In order to convert the tests to parameterized tests, the test case for the client ID was changed: instead of using get_random_bytes to generate the client ID, the client ID is now hardcoded in the test case. >>> >>> Generally "random" usage is not incompatible with parameterized tests, we >>> can >>> create parameterized tests that use random data. >>> The idea is to pass a function that generates the actual param (where we >>> have a >>> pointer to function as one of the members in "params" struct). >>> >>> For example, see "random_dp_query_enc_client_id" usage here: >>> https://lore.kernel.org/dri-devel/20220117232259.180459-7-michal.winiar...@intel.com/ Although it is possible, I don't see the benefit in this case to use the get_random_bytes instead of hardcoding. I believe it will only add more boilerplate to the tests. >>> >>> In this case, we just compare data going in with data going out (and the >>> data >>> itself is not transformed in any way), so it doesn't really matter for >>> coverage >>> and we can hardcode. >>> >>> -Michał >> >> FWIW, while the uses of randomness in DRM tests so far haven't >> concerned me much, I think we'll eventually want to have some way of >> ensuring the inputs to tests are deterministic. >> >> My thoughts are that (at some point) we'll add a kunit_random() >> function or similar, which will use a pseudorandom number generator >> which can be set to a deterministic seed before each test case. That >> way, there'd be a way to reproduce an error easily if it occurred. (Of >> course, there'd be a way of setting different or random seeds to >> preserve the extra coverage you'd otherwise get.) > > That's exactly what DRM tests do (well... most DRM tests, this one being the > exception, and those other tests also seem to have lost a printk with seed > value > after being refactored into kunit). I will take a look at those lost printk in drm_mm_test and drm_buddy_test, as they are important to assure the reproducibility of the tests. > See the usage of DRM_RND_STATE in drm_mm_test and drm_buddy_test. > Having kunit_random() would definitely be useful and let us remove bunch of > boilerplate from the tests, but it doesn't prevent using reproducible random > data in existing tests. > >> I don't think this is something worth holding up or changing existing >> tests at the moment, but having tests behave deterministically is >> definitely desirable, so +1 to avoiding get_random_bytes() if it's not >> giving you any real benefit. > > Yeah - all I was refering to in my previous message was the wording of the > commit message. We're just removing it because it is desirable in this > particular case, not because of the fact that the test is now parameterized > and > that's somehow preventing get_random_bytes() usage. I will send a v3 rewording the commit message to make it more clear. Best Regards, - Maíra Canal > > -Michał > >> We've also had a few requests in the past for being able to pass in a >> custom set of parameters from userspace, which opens up some other >> interesting possibilities, though it's not a priority at the moment. >> >> Cheers, >> -- David > >
Re: [PATCH v2 2/2] drm/tests: Split drm_test_dp_mst_sideband_msg_req_decode into parameterized tests
On Fri, Sep 30, 2022 at 02:50:43PM +0800, David Gow wrote: > On Fri, Sep 30, 2022 at 6:33 AM Michał Winiarski > wrote: > > > > On Tue, Sep 27, 2022 at 07:12:06PM -0300, Maíra Canal wrote: > > > The drm_test_dp_mst_sideband_msg_req_decode repeats the same test > > > structure with different parameters. This could be better represented > > > by parameterized tests, provided by KUnit. > > > > > > In order to convert the tests to parameterized tests, the test case for > > > the client ID was changed: instead of using get_random_bytes to generate > > > the client ID, the client ID is now hardcoded in the test case. > > > > Generally "random" usage is not incompatible with parameterized tests, we > > can > > create parameterized tests that use random data. > > The idea is to pass a function that generates the actual param (where we > > have a > > pointer to function as one of the members in "params" struct). > > > > For example, see "random_dp_query_enc_client_id" usage here: > > https://lore.kernel.org/dri-devel/20220117232259.180459-7-michal.winiar...@intel.com/ > > > > In this case, we just compare data going in with data going out (and the > > data > > itself is not transformed in any way), so it doesn't really matter for > > coverage > > and we can hardcode. > > > > -Michał > > FWIW, while the uses of randomness in DRM tests so far haven't > concerned me much, I think we'll eventually want to have some way of > ensuring the inputs to tests are deterministic. > > My thoughts are that (at some point) we'll add a kunit_random() > function or similar, which will use a pseudorandom number generator > which can be set to a deterministic seed before each test case. That > way, there'd be a way to reproduce an error easily if it occurred. (Of > course, there'd be a way of setting different or random seeds to > preserve the extra coverage you'd otherwise get.) That's exactly what DRM tests do (well... most DRM tests, this one being the exception, and those other tests also seem to have lost a printk with seed value after being refactored into kunit). See the usage of DRM_RND_STATE in drm_mm_test and drm_buddy_test. Having kunit_random() would definitely be useful and let us remove bunch of boilerplate from the tests, but it doesn't prevent using reproducible random data in existing tests. > I don't think this is something worth holding up or changing existing > tests at the moment, but having tests behave deterministically is > definitely desirable, so +1 to avoiding get_random_bytes() if it's not > giving you any real benefit. Yeah - all I was refering to in my previous message was the wording of the commit message. We're just removing it because it is desirable in this particular case, not because of the fact that the test is now parameterized and that's somehow preventing get_random_bytes() usage. -Michał > We've also had a few requests in the past for being able to pass in a > custom set of parameters from userspace, which opens up some other > interesting possibilities, though it's not a priority at the moment. > > Cheers, > -- David
Re: [PATCH v2 2/2] drm/tests: Split drm_test_dp_mst_sideband_msg_req_decode into parameterized tests
On Fri, Sep 30, 2022 at 6:33 AM Michał Winiarski wrote: > > On Tue, Sep 27, 2022 at 07:12:06PM -0300, Maíra Canal wrote: > > The drm_test_dp_mst_sideband_msg_req_decode repeats the same test > > structure with different parameters. This could be better represented > > by parameterized tests, provided by KUnit. > > > > In order to convert the tests to parameterized tests, the test case for > > the client ID was changed: instead of using get_random_bytes to generate > > the client ID, the client ID is now hardcoded in the test case. > > Generally "random" usage is not incompatible with parameterized tests, we can > create parameterized tests that use random data. > The idea is to pass a function that generates the actual param (where we have > a > pointer to function as one of the members in "params" struct). > > For example, see "random_dp_query_enc_client_id" usage here: > https://lore.kernel.org/dri-devel/20220117232259.180459-7-michal.winiar...@intel.com/ > > In this case, we just compare data going in with data going out (and the data > itself is not transformed in any way), so it doesn't really matter for > coverage > and we can hardcode. > > -Michał FWIW, while the uses of randomness in DRM tests so far haven't concerned me much, I think we'll eventually want to have some way of ensuring the inputs to tests are deterministic. My thoughts are that (at some point) we'll add a kunit_random() function or similar, which will use a pseudorandom number generator which can be set to a deterministic seed before each test case. That way, there'd be a way to reproduce an error easily if it occurred. (Of course, there'd be a way of setting different or random seeds to preserve the extra coverage you'd otherwise get.) I don't think this is something worth holding up or changing existing tests at the moment, but having tests behave deterministically is definitely desirable, so +1 to avoiding get_random_bytes() if it's not giving you any real benefit. We've also had a few requests in the past for being able to pass in a custom set of parameters from userspace, which opens up some other interesting possibilities, though it's not a priority at the moment. Cheers, -- David smime.p7s Description: S/MIME Cryptographic Signature
Re: [PATCH v2 2/2] drm/tests: Split drm_test_dp_mst_sideband_msg_req_decode into parameterized tests
On Tue, Sep 27, 2022 at 07:12:06PM -0300, Maíra Canal wrote: > The drm_test_dp_mst_sideband_msg_req_decode repeats the same test > structure with different parameters. This could be better represented > by parameterized tests, provided by KUnit. > > In order to convert the tests to parameterized tests, the test case for > the client ID was changed: instead of using get_random_bytes to generate > the client ID, the client ID is now hardcoded in the test case. Generally "random" usage is not incompatible with parameterized tests, we can create parameterized tests that use random data. The idea is to pass a function that generates the actual param (where we have a pointer to function as one of the members in "params" struct). For example, see "random_dp_query_enc_client_id" usage here: https://lore.kernel.org/dri-devel/20220117232259.180459-7-michal.winiar...@intel.com/ In this case, we just compare data going in with data going out (and the data itself is not transformed in any way), so it doesn't really matter for coverage and we can hardcode. -Michał > So, convert drm_test_dp_mst_sideband_msg_req_decode into parameterized > tests and make the tests' allocations and prints completely managed by KUnit. > > Signed-off-by: Maíra Canal > --- > v1 -> v2: > https://lore.kernel.org/dri-devel/20220925222719.345424-1-mca...@igalia.com/T/#m056610a23a63109484afeafefb5846178c4d36b2 > - Mention on the commit message the change on the test case for the client ID > (Michał Winiarski). > --- > .../gpu/drm/tests/drm_dp_mst_helper_test.c| 370 -- > 1 file changed, 243 insertions(+), 127 deletions(-)
