zabetak commented on PR #3145: URL: https://github.com/apache/calcite/pull/3145#issuecomment-1590951163
@mihaibudiu In Calcite we haven't used Dynamic Tests so far but we do make use of [Parameterized Tests](https://junit.org/junit5/docs/current/user-guide/#writing-tests-parameterized-tests). The two kinds of tests are very similar so I am fine with any of them; whatever is most convenient for you. Although a new release may bring in some good enhancements I am not sure if it is strictly mandatory at this point. We know exactly the test files so we could hard code those here and generate a dynamic test for each. > Rather than checking that the number of failures doesn't change, it's better to check that there are no failures other than the list of 'expected failures' I will try to elaborate a bit more on the comment by Julian. I think what he had in mind can achieved with the following setup. For each test file that we decide to run we can create a "golden" file under `src/test/resources/sqllogictest` with the known/expected failures. Once we run the test we create the "actual" file with the failures and compare "golden" with "actual"; if they differ the test fails otherwise it passes. This is very much inline with how `DiffRepository` works so you could try to take advantage of it in these tests. If `DiffRepository` is not pertinent then you could opt for a simple diff or plain string equality comparison between the files. However, given that we have a large number of failures it might be impractical to create 622 golden files cause we might bloat the repository with test files containing errors. Currently everything under `src/test/resources` (data, golden files, logging configs, etc.) is ~3.5MB. ``` find . -wholename "*/src/test/resources/*" -exec du -ch {} + | grep total ``` If the sqllogic golden files containing the errors are around ~10MB it would be acceptable but going much more beyond that can be problematic and we should probably rethink this. Using compression to keep the file size low should be fine I guess. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
