LuciferYang commented on PR #58513: URL: https://github.com/apache/spark/pull/58513#issuecomment-5540391742
@peter-toth, two questions about how much further either fix should travel, both of which look like committer calls rather than mine. **This one, below 4.3.** The bug is old: positional matching has resolved a column against the projection since 3.2.0 (SPARK-34365), so every maintained line returns wrong values for a pruned positional read. Going below 4.3 is smaller than this PR, not larger: there is no gate to retire, `docs/sql-performance-tuning.md` has no "Merging Subplans" section before 4.3, and `AvroFileFormat.readArchive` does not exist either, so the deserializer change lands with two call sites instead of three. `AvroDeserializer`'s positional path and `AvroUtils.AvroSchemaHelper` are the same shape on 3.5 through 4.2. Say the word and I will open them; `branch-3.3` and `branch-3.4` look retired, last commits 2024-03 and 2024-10. **SPARK-59107, below 4.3.** That one reproduces there too. Measured with two csv shapes: on `branch-4.2`, `DROPMALFORMED` answers `[8, 80]` against `[10, 80]` with `MergeSubplans` excluded and `PERMISSIVE` with `_corrupt_record` answers `[1, 80]` against `[0, 80]`; on `branch-3.5`, where the rule is `MergeScalarSubqueries`, the two-subquery query answers `[8, 80]` and `[1, 80]` while the first subquery run on its own answers `10` and `0`. `branch-4.0` and `branch-4.1` carry 3.5's rule and were not run. The fix does not port as written, though. From 4.2 down the rule lives in `sql/catalyst` (`optimizer/MergeSubplans.scala` on 4.2, `optimizer/MergeScalarSubqueries.scala` on 3.5 through 4.1), while the predicate it needs is about `HadoopFsRelation` and the `FileFormat` subclasses, all of which are in `sql/core`; catalyst references none of them today. So it would need a new seam on the catalyst side, something like a trait catalyst defines and `LogicalRelation` implements by asking the relation. That is a structural change on four maintenance branches, which is why I would rather ask than write it: worth doing, or better recorded as a known limitation on SPARK-59107 with the fix starting at 4.3? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
