Awesome, I will submit the review request in a few minutes. Cheers, Abdullah.
Amoudi, Abdullah. On Thu, Dec 3, 2015 at 2:38 PM, Chris Hillery <[email protected]> wrote: > Yes, please propose the change. I've been looking at overhauling the test > framework as well so I will review. > > For Zorba, I implemented a "known failing" mechanism that allowed you to > mark a test that was currently broken (associated with a ticket ID) without > disabling it. The framework would continue to execute it and expect it to > fail. It would also cause the test run to fail if the test started to > succeed (ie, the bug was fixed) which ensured that the "known failing" mark > would get removed in a timely fashion. To be clear, this is completely > distinct from a negative test case - it was a way to not worry about > forgetting tests that had to be disabled due to known bugs, and to ensure > that all such known bugs had an associated tracking ticket. It was quite > useful there and I was planning to re-introduce it here. > > Ceej > aka Chris Hillery > > On Thu, Dec 3, 2015 at 2:29 PM, abdullah alamoudi <[email protected]> > wrote: > > > Hi All, > > Today, I implemented a fix for a critical issue that we have and wanted > to > > add a new kind of test cases where the test case has 3 files: > > > > 1. Creating the dataset. > > 2. Fill it with data that have duplicate keys. This is expected to throw > a > > duplicate key exception. > > 3. Delete the dataset. This is expected to pass (the bug was here where > it > > is not being deleted). > > > > With the current way we use the test framework, we are unable to test > such > > case and so I started to improve the test framework starting with > actually > > checking the type of exception thrown and making sure that it matches the > > expected error. > > > > ... and boom. I found that many test cases fail but nobody notices > because > > no one checks the type of exception thrown. Moreover, If a test is > expected > > to fail and it doesn't, the framework doesn't check for that. In > addition, > > sometimes the returned exception is meaningless and that is something we > > absolutely must avoid. > > > > What I propose is that I push to master the improved test framework and > > disable the failing test cases, create JIRA issues for them and assign > each > > to someone to look at them. > > > > Thoughts? > > > > Amoudi, Abdullah. > > >
