Thanks Damon for the PR. I'm excited about this development. I'm in favor of minimal dependencies and propose that we stick with ubuntu base images. I've had better success installing maven and other tools individually rather than using the blotted container images. With layering, those tools won't be installed again, so it's fine. I will share some more comments in the PR.
---- On Wed, 07 Dec 2022 14:49:43 -0800 Damon Cortesi <dac...@apache.org> wrote --- Appreciate the feedback! Regarding the first two points, I think we'll learn more as we begin to modernize the project including deprecation of Python2 and moving to a newer version of Java. The maven/alpine images are nice because they've got two crucial build dependencies easily pinned, whereas with Ubuntu those versions are more static but that doesn't preclude us from downloading a specific version if need be. > Wouldn't publishing it as an image make it harder to make changes than just > changing the dockerfile Yes, ineed. For now, I think the Dockerfile will just be used locally for build/test with a known-good environment. It could be nice to provide these images on github so folks don't need to build them themselves...but certainly not a requirement. Thanks! Damon On 2022/12/07 22:40:44 larry mccay wrote: > Hi Damon - > > I saw the PR - that's great! > > - Is it important to have the Dockerfile as close to travis as possible > I could see how this could be meaningful but I don't consider it a must. > > - Will alpine-based images be tough to maintain/support > I lack whatever familiarity I need to contribute to this question. > > - Will we publish some version of the resulting images, if so size could be > important > I don't imagine that we would but perhaps I am thinking about the images > differently. > Do you mean the local image that is created when spinning up the build > environment? > Wouldn't publishing it as an image make it harder to make changes than just > changing the dockerfile, etc and letting it be realized on local dev > machines? > Would this be more for releng orgs that need to build as well? > > thanks! > > --larry > > On Wed, Dec 7, 2022 at 5:32 PM Damon Cortesi <mailto:dac...@apache.org> > wrote: > > > I've got an initial PR for this here: > > https://github.com/apache/incubator-livy/pull/367 > > > > It uses Ubuntu Xenial (similar to our Travis environment) and installs the > > necessary Python packages and R. I have not added Spark to the package as > > it gets pulled down as a dependency, but am considering doing that since > > it's such a large download. > > > > I've got another version of the Dockerfile based off of > > "maven:3-jdk-8-alpine" that works as well. It's quite a bit smaller than > > the xenial version (521MB vs. 1.18GB), but still figuring out exactly how I > > want the Docker image to work. A couple open questions > > > > - Is it important to have the Dockerfile as close to travis as possible > > - Will alpine-based images be tough to maintain/support > > - Will we publish some version of the resulting images, if so size could > > be important > > > > Damon > > > > On 2022/12/03 01:20:45 Damon Cortesi wrote: > > > Coming back to this as I get my dev environment up and running, there's > > definitely an intermix of dependencies between Spark, Python, and R that > > I'm still working out. > > > > > > For example, when I try to start sparkR I get an error message that > > "package ‘SparkR’ was built under R version 4.0.4", but locally I have R > > version 3.5.2 installed. Spark 3.3.1 says you need R 3.5+. That said, think > > my version of R works with Spark2 (at least the tests indicate that...) > > > > > > It'd be great to have a minimum viable environment with specific > > versions and I hope to have that in a Docker environment by early next > > week. :) > > > > > > Currently I'm just basing it off a debian image with Java8, although > > there are Spark images that could be useful... > > > > > > Damon > > > > > > On 2022/11/20 18:55:35 larry mccay wrote: > > > > Considering there is no download for anything older than 3.2.x on the > > > > referred download page, we likely need some change to the README.md to > > > > reflect a more modern version. > > > > We also need more explicit instructions for installing Spark than just > > the > > > > download. Whether we detail this or point to Spark docs that are > > sufficient > > > > is certainly a consideration. > > > > > > > > At the end of the day, we are missing any sort of quick start guide for > > > > devs to be able to successfully build and/or run tests. > > > > > > > > Thoughts? > > > > > > > > On Sat, Nov 19, 2022 at 6:23 PM larry mccay > > > > <mailto:larry.mc...@gmail.com> > > wrote: > > > > > > > > > Hey Folks - > > > > > > > > > > Our Livy README.md indicates the following: > > > > > > > > > > To run Livy, you will also need a Spark installation. You can get > > Spark > > > > > releases at https://spark.apache.org/downloads.html. > > > > > > > > > > Livy requires Spark 2.4+. You can switch to a different version of > > Spark > > > > > by setting the SPARK_HOME environment variable in the Livy server > > > > > process, without needing to rebuild Livy. > > > > > > > > > > Do we have any variation on this setup at this point in the real > > world? > > > > > > > > > > What do your dev environments actually look like and how are you > > > > > installing what versions of Spark as a dependency? > > > > > > > > > > Thanks! > > > > > > > > > > --larry > > > > > > > > > > > > > > >
