LuciferYang commented on PR #58411: URL: https://github.com/apache/spark/pull/58411#issuecomment-5470560153
Thanks for re-running it, @peter-toth. Both new items are in at `efdd55d4f13`. **10.** I kept the test and kept a value assertion with it, by taking the expected row from the same query with `MergeSubplans` excluded instead of from a literal `[10, 10]`. It then says what this PR guarantees, that merging does not change the answer, and it survives the `[10, 100]` you measured on both sides at #58409's head. With `isProjectionSensitiveRead` forced to false it fails with correct answer `[10, 10]` against Spark answer `[10, 100]`, which also rules out a vacuous comparison: one taken from a still-merged run would pass. It asserts the scan columns as well, with `spark.sql.adaptive.enabled` off as you said, and I checked that with it on the collection is empty rather than merely different. The two strictness flags are pinned rather than inherited, so positional matching is the only thing making that read projection-sensitive. The order is now in all three descriptions: this one, then #58340, then #58409, which deletes the avro arm, the `org.apache.spark.sql.avro` import `DataSourceUtils` needs for it, and this test. It only matters because a branch cannot delete code that is not yet in master; with the assertions above, neither direction turns anything red. #58340's `AvroV2Suite` case needs nothing: it asserts the capability, the withholding under the option, and a data query with the option off, and none of the three depends on how a pruned positional read resolves. **11.** Dropped. 210 is the only construction site and the other two are `mp.copy`, so nothing else moved. **4.** Agreed, no configuration. If a third-party format ever needs one, the `FileFormat` method you describe is where it belongs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
