Github user vackosar commented on the issue:
https://github.com/apache/spark/pull/21919
@jose-torres thx for good point. The reason for placing this into
WriterCommitMessage is to set a standard information that should passed at the
commit time.
But I agree that row counting specifically could be moved to e.g.
DataWritingSparkTask#run by adding some extension of WriterCommitMessage. There
will however be metrics which wont be possible to move there for example
Minimum and Maximum Offset written
[SPARK-24647](https://issues.apache.org/jira/browse/SPARK-24647)
Do you agree with extending WriterCommitMessage and using in
DataWritingSparkTask to return row count?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
