Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15102
> I agree that if/when we add that ability to add existing partitions
midstream we'd probably need to add two offsets in to the SQL offset for new
partitions.
It's not just existing partitions. If you have a low-value high-volume
stream (which is the kind of situation where you'd want auto offset reset
latest to begin with), you may not even want your first batch to have however
many messages got in between creation and subscription rebalance. I dunno, I
just don't want to assume too much.
> I'd also support JSON here, but I would not mandate it (i.e. try json
parsing and fall back to comma separation). Its not ambiguous, supports
consistent usage, and doesn't penalize the simple use cases.
Cool, seems reasonable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
