I would suggest to put the binaries as download files in our systemds webpage.
This makes it robust since we have full control over the files, and removes them from the repository. Then upon building the system we would download them from that source. The idea could also be applied to small datasets that we want to provide in tests and in tutorial examples. Best regards Sebastian ________________________________ From: Matthias Boehm <[email protected]> Sent: Thursday, February 25, 2021 5:42:18 PM To: [email protected] Subject: Re: Binaries in the repository thanks for starting this discussion. Personally, I would prefer keeping them in the repo, even if partially outdated during development cycles. However, building SystemDS should always pack these binaries into the self-contained SystemDS.jar to avoid unnecessary friction and unexpected behavior. While we can download them from somewhere during the build, it would require updating that external download source, and some robust build integration such that builds can still be done offline. Regards, Matthias On 2/25/2021 2:03 PM, Mark Dokter wrote: > Hi! > > Frequent changing binaries in a git repo are a pain and a known problem > [1]. Every change I make to the native code would require changed > binaries. What's even more annoying is booting into Windows or Linux to > also provide the binaries for "the other" (depending on where I work > atm). The latter is actually a good thing because it forces me to check > if everything's working out on the other platform frequently. > > There are several methods/projects dealing with binaries [2]. I didn't > thoroughly investigate but maybe we'll have something like that in the > future. > > The requirement to have binaries is there because we want to have easy > access to native and gpu operations for our users. Going by that reason > I argue that users would take a binary release to start out with > SystemDS. This is why I suggest to reduce the publishing of binary files > to release zips/tars. > > With an up to date documentation (mental note to myself), developers who > work with the sources directly can be expected to be able to install > dependencies and use some simple steps with cmake (yes we can even put > that in a script). > > rfc, > Mark > > > [1] > https://stackoverflow.com/questions/4697216/is-git-good-with-binary-files > > [2] > https://stackoverflow.com/questions/540535/managing-large-binary-files-with-git/29530784#29530784 >
