well, as there is not much harm adding further examples (incl docker
images), I could be persuaded to support it.
However, let's please be very careful to make clear this is not
required, nor the recommended way of running SystemDS because such
images are an additional level of indirection. In local setups, the only
thing needed is Java, our jar, and the libraries we depend on (packaged
into our distribution) and on Win winutils in an appropriate environment
variable; in a distributed Spark environment, only our jar file is
needed. All other things like native BLAS libraries and GPU support, are
relevant for performance not prototyping or trying things out.
Regards,
Matthias
On 5/10/2021 11:29 PM, Janardhan wrote:
What is "the burden of individual component installation" - isn't it a
git clone and mvn package?
There are a number of distro's which do not distribute Java with it, some
times maven. For us, these two steps look obvious even these steps are
complicated from a user point of view. When our team is onboarded, most
of the time I personally help them install maven & dependencies. :)
I have never succeeded in compiling our Makefiles or running .PTX always
some problems. Tried it on Windows, WSL, Ubuntu, CentOS, even on brand
new VMs on cloud - yes, I did put in effort but the day 2 comes.
Many servers in an office setting now come with docker installed, so it is
only
one familiar command.
` docker run ...`
Also, are you sure that the GPU would not be able to pass through to a
docker instance on a Windows machine?
AFAIK, the nvidia docker[1] does not support anything other than linux.
But, I think it is possible - we can explore this option later. :)
What's next?
- We will stick to CPU images for the coming release with base image
tags - latest, 2.1.0, dev
- If the time permits for testing a basic variant with GPU support.
[1] https://github.com/NVIDIA/nvidia-docker/wiki#platform-support
Thank you,
Janardhan
On Mon, May 10, 2021 at 3:28 PM Baunsgaard, Sebastian
<baunsga...@tugraz.at.invalid> wrote:
I think that the docker images sound like a good idea,
especially because of setup difficulties, i can imagine that simply using
an
image with preinstalled BLAS or MKL and GPU drivers would be much cleaner
for beginner setup and execution than having people find and install all
of these.
That said someone have to setup these images, and preferably automate the
building like done in Apache Airflow.
Also, are you sure that the GPU would not be able to pass through to a
docker instance on a Windows machine?
https://www.tensorflow.org/install/docker?hl=nb
________________________________
From: Matthias Boehm <mboe...@gmail.com>
Sent: Monday, May 10, 2021 11:49:14 AM
To: dev@systemds.apache.org
Subject: Re: [DISCUSS][SYSTEMDS-2974] Using SystemDS with Docker
personally I don't see a good use case for such docker images. We use
docker for the test setup including installed R packages, but other than
that try to keep the setup as lean as possible (which is important given
the growing number of APIs and deployment environments).
What is "the burden of individual component installation" - isn't it a
git clone and mvn package?
Regards,
Matthias
On 5/10/2021 4:30 AM, Janardhan wrote:
Hi all,
We already have docker support for testing, the same files
can be utilized for a public image (with CPU, GPU and Jupyter).
Use cases:
1. Invoke dml or python scripts right inside container
2. Prototype algorithm/pipelines without the burden of individual
component installation.
3. Working with GPU (only Linux supported)
Anyone would want to volunteer for the implementation of the same,
the intended docker support is documented at [1].
How to implement:
1. Build image tags for release channels (latest, v1.2.0, dev).
2. For GPU follow the instructions in [1]. Can be implemented later.
[1] https://github.com/apache/systemds/pull/1271
Thank you,
Janardhan
