We will keep your comments into account. Feel free to make comments on the PRs in case there is some deviation.
Thanks, Janardhan On Tue, May 11, 2021 at 3:14 AM Matthias Boehm <mboe...@gmail.com> wrote: > well, as there is not much harm adding further examples (incl docker > images), I could be persuaded to support it. > > However, let's please be very careful to make clear this is not > required, nor the recommended way of running SystemDS because such > images are an additional level of indirection. In local setups, the only > thing needed is Java, our jar, and the libraries we depend on (packaged > into our distribution) and on Win winutils in an appropriate environment > variable; in a distributed Spark environment, only our jar file is > needed. All other things like native BLAS libraries and GPU support, are > relevant for performance not prototyping or trying things out. > > Regards, > Matthias > > On 5/10/2021 11:29 PM, Janardhan wrote: > >> What is "the burden of individual component installation" - isn't it a > >> git clone and mvn package? > > > > There are a number of distro's which do not distribute Java with it, some > > times maven. For us, these two steps look obvious even these steps are > > complicated from a user point of view. When our team is onboarded, most > > of the time I personally help them install maven & dependencies. :) > > > > I have never succeeded in compiling our Makefiles or running .PTX always > > some problems. Tried it on Windows, WSL, Ubuntu, CentOS, even on brand > > new VMs on cloud - yes, I did put in effort but the day 2 comes. > > > > Many servers in an office setting now come with docker installed, so it > is > > only > > one familiar command. > > > > ` docker run ...` > > > >> Also, are you sure that the GPU would not be able to pass through to a > >> docker instance on a Windows machine? > > > > AFAIK, the nvidia docker[1] does not support anything other than linux. > > But, I think it is possible - we can explore this option later. :) > > > > What's next? > > > > - We will stick to CPU images for the coming release with base image > > tags - latest, 2.1.0, dev > > - If the time permits for testing a basic variant with GPU support. > > > > [1] https://github.com/NVIDIA/nvidia-docker/wiki#platform-support > > > > Thank you, > > Janardhan > > > > On Mon, May 10, 2021 at 3:28 PM Baunsgaard, Sebastian > > <baunsga...@tugraz.at.invalid> wrote: > > > >> I think that the docker images sound like a good idea, > >> > >> especially because of setup difficulties, i can imagine that simply > using > >> an > >> > >> image with preinstalled BLAS or MKL and GPU drivers would be much > cleaner > >> > >> for beginner setup and execution than having people find and install all > >> of these. > >> > >> > >> That said someone have to setup these images, and preferably automate > the > >> > >> building like done in Apache Airflow. > >> > >> > >> Also, are you sure that the GPU would not be able to pass through to a > >> docker instance on a Windows machine? > >> > >> https://www.tensorflow.org/install/docker?hl=nb > >> > >> > >> > >> > >> ________________________________ > >> From: Matthias Boehm <mboe...@gmail.com> > >> Sent: Monday, May 10, 2021 11:49:14 AM > >> To: dev@systemds.apache.org > >> Subject: Re: [DISCUSS][SYSTEMDS-2974] Using SystemDS with Docker > >> > >> personally I don't see a good use case for such docker images. We use > >> docker for the test setup including installed R packages, but other than > >> that try to keep the setup as lean as possible (which is important given > >> the growing number of APIs and deployment environments). > >> > >> What is "the burden of individual component installation" - isn't it a > >> git clone and mvn package? > >> > >> Regards, > >> Matthias > >> > >> On 5/10/2021 4:30 AM, Janardhan wrote: > >>> Hi all, > >>> > >>> We already have docker support for testing, the same files > >>> can be utilized for a public image (with CPU, GPU and Jupyter). > >>> > >>> Use cases: > >>> > >>> 1. Invoke dml or python scripts right inside container > >>> 2. Prototype algorithm/pipelines without the burden of individual > >>> component installation. > >>> 3. Working with GPU (only Linux supported) > >>> > >>> Anyone would want to volunteer for the implementation of the same, > >>> the intended docker support is documented at [1]. > >>> > >>> How to implement: > >>> > >>> 1. Build image tags for release channels (latest, v1.2.0, dev). > >>> 2. For GPU follow the instructions in [1]. Can be implemented later. > >>> > >>> [1] https://github.com/apache/systemds/pull/1271 > >>> > >>> Thank you, > >>> Janardhan > >>> > >> > > >
