COLDMANYU opened a new pull request, #10824: URL: https://github.com/apache/paimon/pull/10824
### Purpose Closes #10773. Spark's CSV source drops lines that are empty or contain only whitespace before parsing them. Paimon's CSV reader reads such a line as a row where every column is NULL. When an existing Spark CSV table is moved to a Paimon CSV Format Table, the same files return extra all-NULL rows. In the case described in the issue, one hour of tab-separated gzip logs returned 6 extra rows. Blank lines cannot be skipped unconditionally. Paimon's own `CsvFormatWriter` writes a row whose values are all NULL as an empty line in a single-column table, and today such a line is read back correctly (see the table in #10773). This PR therefore adds an opt-in option: ``` csv.skip-empty-lines = false (default: current behavior) ``` When it is enabled, `CsvFileReader` skips a line if nothing is left after `String#trim`. Spark uses the same test (`CSVExprUtils.filterCommentAndEmpty`: `iter.filter(_.trim.nonEmpty)`), so tables moved from Spark return the same rows. Notes: * The issue suggested `StringUtils#isNullOrWhitespaceOnly`. I used `String#trim` instead so that the result matches Spark exactly. The two only differ for a few characters. `String#trim` also removes control characters up to U+0020, so a line holding only Hive's default delimiter `\u0001` (an all-NULL row) is skipped, as in Spark. Unicode whitespace above U+0020, such as U+3000, is kept. I'm happy to switch if you prefer the other one. * The default behavior does not change. When the option is disabled, nothing is skipped. * The header line (`csv.include-header`) is still handled before any line is skipped. Splitting by offset and length is not affected, because skipping happens per line after the line reader. * The option is documented in the CSV format options table. ### Tests New `CsvSkipEmptyLinesTest`: * `testDefaultReadsBlankLinesAsNullRows`: without the option, empty, space-only and tab-only lines are still read as all-NULL rows. * `testSkipEmptyLines`: with the option, these lines are skipped and all other rows are returned in order. * `testSkipEmptyLinesInGzipFile`: the same for a gzip file. * `testSkipEmptyLinesWithHeader`: with `csv.include-header`, the header is still skipped and blank lines after it are skipped too. * `testSkipEmptyLinesWithHiveDefaultDelimiter`: a line holding only `\u0001` is skipped, as in Spark. The four tests that enable the option fail without the change in `CsvFileReader`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
