Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20386
I like this change! It adds a missing feature which is required for
migrating the file-based data source(which use `FileCommitProtocol` and has a
callback for task commit), and also make it possible to handle large jobs,
which have a lot of tasks. Implementations can externalize the commit messages
to avoid keeping too many messages in memory.
LGTM, waiting feedback from others.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
