Joorgem commented on PR #57608: URL: https://github.com/apache/spark/pull/57608#issuecomment-5151978306
You're right, and I checked it rather than take it on trust — applied in bc5be21. `multiLine` now uses your wording verbatim. I confirmed the mechanism you describe: with a 3-column schema and `1,"foo\nbar,baz,qux"` the trailing half comes back as a valid `[bar, baz, qux"]`, and with a 2-column schema `1,"hello\nworld"` the *leading* half comes back as a clean `[1, hello]` with no corrupt signal at all. So "both halves are malformed" was wrong, and token count is what decides it, not the unclosed quote. For the `mode` sentence I went slightly further than "some or all", because that second case turned out to be the sharper one and it inverts what the original text claimed: | schema | input | leading half | trailing half | `DROPMALFORMED` keeps | |---|---|---|---|---| | 2 col | `1,"hello⏎world"` | `[1, hello]` — **clean** | malformed | **the truncated row** | | 3 col | `1,"foo⏎bar,baz,qux"` | malformed | `[bar, baz, qux"]` — **clean** | the garbage row | | 3 col | `1,"a⏎b",c` | malformed | malformed | neither | `DROPMALFORMED` dropping *both* halves is the case where nothing silently wrong survives. The one worth warning about is the opposite: a half whose token count happens to match the schema is retained as a valid row carrying a truncated value, and `DROPMALFORMED` keeps it precisely because it is not malformed. The new sentence says that instead of the old absolute. Happy to trim it back to your shorter phrasing if you'd rather keep the entry brief — the correction to the absolute claim is the part that matters, and that is in either way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
