My current fix is to remove the specific lists. (A test which is flaky isn't very valuable.) You can see my current branch at [1]. You can look at my most recent commit to see the changes I made to the expected results files to get the tests to report the correct result.
If you want to do file name verification (as opposed to simply numFiles verification), it seems like you should be doing a structured verification of the json plan rather than trying to do something with the text plan. [1] https://github.com/dremio/drill-test-framework -- Jacques Nadeau CTO and Co-Founder, Dremio On Mon, Oct 26, 2015 at 9:36 AM, Chun Chang <[email protected]> wrote: > Jacques, > > Thanks for the heads up. And what is your proposed fix? I think we can > easily enhance the test framework to handle file ordering cases. This way, > we don't need to modify any existing tests. > > Thanks, > Chun > > On Sat, Oct 24, 2015 at 9:11 PM, Jacques Nadeau <[email protected]> > wrote: > > > A large number of tests associated with MD-185 are unintentionally > brittle. > > Many of these tests reference more than one file. The tests are planning > > tests and thus are prone to failure if the file ordering isn't the same > as > > what was used for test generation. > > > > In the case of these types of tests, people should be cautious of adding > > expected results that include multiple files in a certain order. I'm > > working on a patch fix for these specific tests but wanted to let people > > know to be cautious of these issues in the future. > > > > thanks! > > > > -- > > Jacques Nadeau > > CTO and Co-Founder, Dremio > > >
