Jose Torres created SPARK-23503:

             Summary: continuous execution should sequence committed epochs
                 Key: SPARK-23503
             Project: Spark
          Issue Type: Sub-task
          Components: Structured Streaming
    Affects Versions: 2.4.0
            Reporter: Jose Torres

Currently, the EpochCoordinator doesn't enforce a commit order. If a message 
for epoch n gets lost in the ether, and epoch n + 1 happens to be ready for 
commit earlier, epoch n + 1 will be committed.


This is either incorrect or needlessly confusing, because it's not safe to 
start from the end offset of epoch n + 1 until epoch n is committed. 
EpochCoordinator should enforce this sequencing.


Note that this is not actually a problem right now, because the commit messages 
go through the same RPC channel from the same place. But we shouldn't 
implicitly bake this assumption in.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to