I noticed that in the main branch, the ec2 directory along with the spark-ec2 script is no longer present.
Is spark-ec2 going away in the next release? If so, what would be the best alternative at that time? A couple more additional questions: 1. Is there any way to add/remove additional workers while the cluster is running without stopping/starting the EC2 cluster? 2. For 1, if no such capability is provided with the current script., do we have to write it ourselves? Or is there any plan in the future to add such functions? 2. In PySpark, is it possible to dynamically change driver/executor memory, number of cores per executor without having to restart it? (e.g. via changing sc configuration or recreating sc?) Our ideal scenario is to keep running PySpark (in our case, as a notebook) and connect/disconnect to any spark clusters on demand.
