LuciferYang opened a new issue, #9555: URL: https://github.com/apache/paimon/issues/9555
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `2788fe596` (2.1-SNAPSHOT). `release-1.4` carries it as well; 1.2 and 1.3 do not. ### Compute Engine Flink and Spark, on any table read through Paimon's own CSV reader (`'file.format' = 'csv'`). For a format table that is the default path, since `format-table.implementation` defaults to `paimon`. ### Minimal reproduce step Read a three-column CSV whose first field does not parse as INT, with the default `csv.mode`: ```sql CREATE TABLE t (a INT, b STRING, c DOUBLE) WITH ( 'type' = 'format-table', 'file.format' = 'csv' ); -- one line in the table directory: x,Alice,1.5 SELECT * FROM t; ``` The row comes back as `(null, null, null)`, although `Alice` and `1.5` are both well formed. `CsvParser.parse` walks the projected fields in a `for` loop, and the PERMISSIVE branch breaks out of that loop: ```java for (int i = 0; i < projectMapping.length; i++) { ... if (parseResult != null && parseResult.getLeft()) { row.setField(i, parseResult.getValue()); } else if (mode == PERMISSIVE && (parseResult == null || !parseResult.getLeft() || exception != null)) { break; } ``` Every field after the malformed one keeps the null that `new GenericRow(...)` started with. ### What doesn't meet your expectations? `csv.mode` defaults to PERMISSIVE, and the option is documented as "sets malformed fields to null", which means the field and not the rest of the row. Nothing throws and nothing is logged, so the query returns wrong data quietly. A row whose malformed field happens to be the last projected column reads correctly, which is why the current tests pass. ### Anything else? This regressed in #6856 (`184b95aae`). The mode used to be handled by a switch: ```java switch (mode) { case PERMISSIVE: break; ``` There the `break` left the switch and the loop went on to the next field. That commit turned the switch into an if-else chain, where the same `break` leaves the loop instead. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
