Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/20552
It's my intent to say that other data sources built by general developers
aren't supposed to use batch ids in the executors for any purpose. In addition
to the issue you mentioned, I don't think there's a compelling reason to do so
in the DataSourceV2 model, and I worry it's easy to write implementations that
seem correct but aren't that way.
Since this interface is still evolving, I think it makes sense to revisit
the question if we notice a scenario where it's infeasible to rewrite a piece
of transactional logic to not use the batch ID in the executor.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
