Hi Matthias, Thanks for the feedback! I'll start working on it. I will keep the old scripts around and add examples for the new AWS deployment model as well.
Best, Badrul On Fri, 1 Dec 2023 at 12:04, Matthias Boehm <mboe...@gmail.com> wrote: > Hi Badrul, > > thanks for the initiative. I think this is worth a try and there is no > harm having such deployment alternatives (and related examples) > available. I personally prefer to reduce indirections though (in order > to simplify debugging and unnecessary dependencies out of the way) and > thus, would appreciate keeping the core aws scripts around. > > Regards, > Matthias > > On 12/1/2023 8:22 PM, Badrul Chowdhury wrote: > > Hi All, > > > > I am writing to solicit your opinion on a proposed change to the way we > > deploy SystemDS to the cloud. > > > > Currently, we use a shell script to deploy Spark with the SystemDS > binaries > > to AWS. The relevant scripts to deploy and teardown reside in > > /Users/badrulchowdhury/code/systemds1/scripts/aws/run_systemds_script.sh. > > Although this works just fine, it has been my experience that it’s very > > brittle with different aws CLI versions. > > > > I am proposing that we switch to using an industry standard framework > such > > as OpenTofu (https://opentofu.org/) to do the same. Doing so would give > us > > much more flexibility to make changes and would make it easier to > maintain > > and change deployment configs using YAML. YAML configs have the added > > advantage of making tracking configs easy using git. Furthermore, > uploading > > new DML files and cleaning up afterwards would become super easy. > > > > We would track all JIRAs for the proposed changes under the existing > > “Infrastructure" component. > > > > > > Would love to hear your thoughts, > > > > Best, > > Badrul > > > > -- Best, Badrul
