[ https://issues.apache.org/jira/browse/SPARK-21621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan resolved SPARK-21621. --------------------------------- Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue resolved by pull request 18830 [https://github.com/apache/spark/pull/18830] > Reset numRecordsWritten after DiskBlockObjectWriter.commitAndGet called > ----------------------------------------------------------------------- > > Key: SPARK-21621 > URL: https://issues.apache.org/jira/browse/SPARK-21621 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.2.0 > Reporter: Xianyang Liu > Fix For: 2.2.1, 2.3.0 > > > We should reset numRecordsWritten to zero after > DiskBlockObjectWriter.commitAndGet called. > Because when `revertPartialWritesAndClose` be called, we decrease the > written records in `ShuffleWriteMetrics` . However, we decreased the written > records to zero, this should be wrong, we should only decreased the number > reords after the last `commitAndGet` called. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org