Hi All, I am writing to solicit your opinion on a proposed change to the way we deploy SystemDS to the cloud.
Currently, we use a shell script to deploy Spark with the SystemDS binaries to AWS. The relevant scripts to deploy and teardown reside in /Users/badrulchowdhury/code/systemds1/scripts/aws/run_systemds_script.sh. Although this works just fine, it has been my experience that it’s very brittle with different aws CLI versions. I am proposing that we switch to using an industry standard framework such as OpenTofu (https://opentofu.org/) to do the same. Doing so would give us much more flexibility to make changes and would make it easier to maintain and change deployment configs using YAML. YAML configs have the added advantage of making tracking configs easy using git. Furthermore, uploading new DML files and cleaning up afterwards would become super easy. We would track all JIRAs for the proposed changes under the existing “Infrastructure" component. Would love to hear your thoughts, Best, Badrul
