Hi everyone, I hope this message finds you well.
We have several use cases involving Spark Structured Streaming that would benefit from auto-scaling. We understand that Dynamic Resource Allocation does not work optimally with Spark Structured Streaming, so we are exploring alternative solutions. Currently, we are monitoring lag and other relevant metrics to determine when to scale our cluster up or down. However, as far as I know, we still need a way to instruct the driver to add new executors and ensure proper communication with them. Is there a recommended approach for achieving this? Thank you very much for your help! Best regards, Nimrod
