Hi All, Last night Abdullah and I merged the Asterix test framework improvement change. So, don’t be surprised if you have a test case that worked yesterday but not on the current master.
During the development of the change, we discovered and fixed critical issues such as: 1. Failed Jobs' threads that are never terminated and their operators keep waiting forever. 2. Many operators that do not comply with the IFrameWriter contract. 3. Indeterministic result delivery on synchronous queries. 4. Invalid number of active operations on an index after a transaction aborts. All of these issues and many others were not detected before because of the state the test framework was in. I encountered test cases that were miraculously passing just because they were expected to throw an exception. Just to give you an example, a test case was using the DDL API to run a query, and an exception was thrown saying "SyntaxError: Invalid statement: Non-Update statement” and it was considered as passing, even though the expected exception was “MetadataException: Unknown dataverse”. Due to those reasons, if you are adding any new test cases, you will have to provide the following: 1. For each query in the test case, an expected result file (even queries that are expected to return no records). 2. For each expected exception in the test case, you have to provide the complete exception type as well as the expected error message. If you have multiple expected exceptions in the same test case, they need to be ordered in the same order they will be raised. Unfortunately, currently we don’t have standard error codes for exceptions (e.g. ASX-0001: “Duplicate key exception”), so any changes to the exceptions messages will need to be reflected on the test cases, but this is better than having a broken test framework. Cheers, Murtadha
