Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21718
It has been fairly easy to rescale partitions before stateful operators
came into play. For structured streaming, it is now not a trivial thing, cause
rescaling partitions should also handle rescaling of state which is stored to
disk. Rescaling state may require reading whole states and redistribute via
hash function, and resave to disk again. That's why SS stores previous conf.
and force using it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
