Hi, Worth checking this link https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation
On Saturday, 28 January 2023 at 06:18:28 GMT, Lingzhe Sun <lingzhe....@hirain.com> wrote: #yiv9684413148 body {line-height:1.5;}#yiv9684413148 ol, #yiv9684413148 ul {margin-top:0px;margin-bottom:0px;list-style-position:inside;}#yiv9684413148 body {font-size:10.5pt;font-family:'Microsoft YaHei UI';color:rgb(0, 0, 0);line-height:1.5;}#yiv9684413148 body {font-size:10.5pt;font-family:'Microsoft YaHei UI';color:rgb(0, 0, 0);line-height:1.5;}Hi all, I'm wondering if dynamic resource allocation works in spark+kafka streaming applications. Here're some questions: - Will structured streaming be supported? - Is the number of consumers always equal to the number of the partitions of subscribed topic (let's say there's only one topic)? - If consumers is evenly distributed across executors, will newly added executor(through dynamic resource allocation) trigger a consumer reassignment? - Would it be simply a bad idea to use dynamic resource allocation in streaming app, because there's no way to scale down number of executors unless no data is coming in? Any thoughts are welcomed. Lingzhe SunHirain Technology