Jason, I misunderstood earlier why unit test is failing. It has nothing to do with ordering of files.
Whats happening is I'm doing topn operation on field which is union of strings and nulls in descending order. Test checks if string values are on top but somehow for some people nulls are on top and test fails. I'm suspecting it has to do with how comparator treats null - high/low. ~ Amit. On Tue, Dec 15, 2015 at 3:23 PM, Jason Altekruse <[email protected]> wrote: > Amit, > > The message out of the test framework tries to provide enough information > to debug even if the issues isn't reproducible in your environment. Can you > think of any reason why it might be giving the different results shown in > the message if the order of the batches changed? > > If you need to change the order yourself there are two hacky approaches you > could do. Try changing the names or saving the files in a different order > to make the FS give them back to you in a different order. You also could > just combine together the files and adjust the batch cutoff number used in > the json reader, with various ordering of the records in different versions > of the dataset. > > As I write this I realize that combining the files will change the behavior > of the read. with the first batch giving a single type and later ones > giving a union type. As opposed to the multiple files approach which would > produce a bunch of different individual types and make the sort operation > generate the union type. To test this properly we may just need a test > harness to produce batches explicitly and feed them into an operator, > rather than relying on the JSON reader. > > - Jason > > On Tue, Dec 15, 2015 at 2:31 PM, Amit Hadke <[email protected]> wrote: > > > Hey Guys, > > > > I'm not able to reproduce same issue and test doesn't seem to be doing > > anything. > > > > Can someone run "mvn -Dtest=TestTopNSchemaChanges#testMissingColumn test" > > and see if it fails? > > > > On Mon, Dec 14, 2015 at 11:51 PM, Amit Hadke <[email protected]> > wrote: > > > > > This seems like a bug in topn code than test. > > > We are expecting sorted by kl2 (descending) so that non null values > come > > > up on top. > > > Results seems to be have nulls on top. > > > > > > ~ Amit. > > > > > > On Mon, Dec 14, 2015 at 11:27 PM, Jason Altekruse < > > > [email protected]> wrote: > > > > > >> Seems weird that the results would be different based on reading > order, > > as > > >> the queries themselves contain an order by. Do we return different > types > > >> out of the sort depending on which schema we get first? Is this > > >> intentional? > > >> > > >> - Jason > > >> > > >> On Mon, Dec 14, 2015 at 6:06 PM, Steven Phillips <[email protected]> > > >> wrote: > > >> > > >> > I just did a build a linux box, and didn't see this failure. My > guess > > is > > >> > that it fails depending on which order the files are read. > > >> > > > >> > On Mon, Dec 14, 2015 at 5:38 PM, Venki Korukanti < > > >> > [email protected]> > > >> > wrote: > > >> > > > >> > > Is anyone else seeing below failure on latest master? I am running > > it > > >> on > > >> > > Linux. > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > testMissingColumn(org.apache.drill.exec.physical.impl.TopN.TestTopNSchemaChanges) > > >> > > Time elapsed: 2.537 sec <<< ERROR! > > >> > > java.lang.Exception: unexpected null at position 0 column '`vl2`' > > >> should > > >> > > have been: 299 > > >> > > > > >> > > Expected Records near verification failure: > > >> > > Record Number: 0 { `kl1` : null,`kl2` : 299,`vl2` : 299,`vl1` : > > >> > null,`vl` : > > >> > > null,`kl` : null, } > > >> > > Record Number: 1 { `kl1` : null,`kl2` : 298,`vl2` : 298,`vl1` : > > >> > null,`vl` : > > >> > > null,`kl` : null, } > > >> > > Record Number: 2 { `kl1` : null,`kl2` : 297,`vl2` : 297,`vl1` : > > >> > null,`vl` : > > >> > > null,`kl` : null, } > > >> > > > > >> > > > > >> > > Actual Records near verification failure: > > >> > > Record Number: 0 { `kl1` : null,`vl2` : null,`kl2` : null,`vl1` : > > >> > null,`vl` > > >> > > : 100.0,`kl` : 100.0, } > > >> > > Record Number: 1 { `kl1` : null,`vl2` : null,`kl2` : null,`vl1` : > > >> > null,`vl` > > >> > > : 101.0,`kl` : 101.0, } > > >> > > Record Number: 2 { `kl1` : null,`vl2` : null,`kl2` : null,`vl1` : > > >> > null,`vl` > > >> > > : 102.0,`kl` : 102.0, } > > >> > > > > >> > > For query: select kl, vl, kl1, vl1, kl2, vl2 from > > >> > > > > >> > > > > >> > > > >> > > > dfs_test.`/root/drill/exec/java-exec/target/1450142361702-0/topn-schemachanges` > > >> > > order by kl2 desc limit 3 > > >> > > at > > >> > > > > >> > > > > >> > > > >> > > > org.apache.drill.DrillTestWrapper.compareValuesErrorOnMismatch(DrillTestWrapper.java:512) > > >> > > at > > >> > > > > >> > > > > >> > > > >> > > > org.apache.drill.DrillTestWrapper.compareMergedVectors(DrillTestWrapper.java:170) > > >> > > at > > >> > > > > >> > > > > >> > > > >> > > > org.apache.drill.DrillTestWrapper.compareMergedOnHeapVectors(DrillTestWrapper.java:397) > > >> > > at > > >> > > > > >> > > > > >> > > > >> > > > org.apache.drill.DrillTestWrapper.compareOrderedResults(DrillTestWrapper.java:352) > > >> > > at > > >> > org.apache.drill.DrillTestWrapper.run(DrillTestWrapper.java:124) > > >> > > at org.apache.drill.TestBuilder.go(TestBuilder.java:129) > > >> > > at > > >> > > > > >> > > > > >> > > > >> > > > org.apache.drill.exec.physical.impl.TopN.TestTopNSchemaChanges.testMissingColumn(TestTopNSchemaChanges.java:206) > > >> > > > > >> > > > > >> > > Results : > > >> > > > > >> > > Tests in error: > > >> > > TestTopNSchemaChanges.testMissingColumn:206 ยป unexpected null > at > > >> > > position 0 c... > > >> > > > > >> > > Tests run: 4, Failures: 0, Errors: 1, Skipped: 0 > > >> > > > > >> > > > >> > > > > > > > > >
