Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/9143#issuecomment-152051139
Discussed offline. Its cleaner to refactor this. Have a
BatchedWritedAheadLog which is a wrapper for other WALs, and is a WAL itself.
Internally it will take care of queueing and batching, and recovering data from
batches. This will be easily unit testable.
The RBT then does not need to change at all, only wrap the WAL in
BatchedWAL when configured.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
