gengliangwang commented on PR #52557: URL: https://github.com/apache/spark/pull/52557#issuecomment-3413874113
@ZiyaZa I am not a big fan of such behavior changes. Also the PR description is a bit confusing to me: > we instead pick an arbitrary field from the Parquet file to read and use that to populate NULLs why just picking one arbitrary field, instead of setting all the fields null? > The heuristic to pick the arbitrary field is as follows: we pick one at the lowest array nesting level (i.e., any scalar field is preferred to array, which is preferred to array<array>) Could you provide more details on this one? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
