JingsongLi commented on code in PR #10824:
URL: https://github.com/apache/paimon/pull/10824#discussion_r4385377522
##########
paimon-format/src/main/java/org/apache/paimon/format/csv/CsvFileReader.java:
##########
@@ -55,7 +57,13 @@ public CsvFileReader(
}
@Override
+ @Nullable
protected InternalRow parseLine(String line) {
+ // Same test as Spark's CSV source (String#trim drops chars <=
U+0020). Returning null
+ // makes the text reader move on to the next line.
+ if (skipEmptyLines && line.trim().isEmpty()) {
Review Comment:
[P2] Keep persisted-table metadata counts consistent when skipping rows
This option is also accepted by a normal append-only FileStoreTable. I
reproduced this with a nullable single INT column, `file.format=csv`,
`file.compression=gzip`, `bucket=-1`, and `csv.skip-empty-lines=true`: write
`(NULL), (1)`, commit, and scan. The reader returns 1 row, but the planned
DataSplit still reports `mergedRowCount() = 2`. With the option disabled, both
counts are 2. Spark's `AggregatePushDownUtils.tryPushdownAggregation` accepts
this split and `CountStarEvaluator` sums that metadata, so a pushed `COUNT(*)`
reports 2 although a normal scan returns only 1. Please restrict this behavior
to external FormatTable reads, or reject/handle it for persisted tables and
disable metadata aggregation wherever rows can be skipped. Add a table-level
regression covering the count versus scanned rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
