All - I have been playing around with a docker based spark environment as described in an article [1]. It isn't clear whether this will work for all tests or the dev experience yet.
I'd like to get some information from others on how they approach these functional dependencies for dev purposes. FYI - for those that don't know, we have a livy channel in the Apache Slack account where we could have some interactive discussion on this and other topics as well. Thanks! --larry 1. https://dev.to/mvillarrealb/creating-a-spark-standalone-cluster-with-docker-and-docker-compose-2021-update-6l4 On Mon, Jan 16, 2023 at 1:48 PM larry mccay <lmc...@apache.org> wrote: > Hey Folks - > > One of the goals for the initial release as captured in > https://issues.apache.org/jira/browse/LIVY-898 is to help clarify the > contribution process. This is to help folks that would like to contribute > to fixing issues and adding features to get started. > > One of the items there is to help describe a dev environment setup. > > I'd love to hear from folks in the community around a couple things: > > 1. What the minimal dev environment would look like preferably with local > deployments > 2. Whether we have different types of dev specializations - such as: > notebook devs, API devs, server, devs, etc - and we could determine whether > we need to represent each or some or some lowest common denominator > 3. Do we use docker for things like spark and yarn dependencies for these > environments > 4. Do we assume all livy contributors have an existing strong background > in spark > > I can say that I am personally used to relying on a remote deployment > within my workplace (usually via VPN) for both Knox and Livy testing. I > don't know whether this is a common dev env scenario or not but it does > seem like it could be a barrier to entry for those that wouldn't have such > a deployment available. > > I think that our recent build improvements with docker have been really > good and am wondering whether a similar approach with system dependencies > would work for dev environments. > > What are your thoughts? > > thanks, > > --larry >
