There is a design document that covers a lot of concerns: https://docs.google.com/document/d/1pcyH5f610X2jyJW9WbWHnj8jktQPLlbbmmUwdeK4fJk, validation included.
We had a discussion about validation (validate before we hit the api server) and was considered too much. In general regarding Rob's options,l I prefer option 3 and that was the design's purpose. For this feature to be completed, it needs to have some minimum functionality implemented. What that might be depends on what you offer. For example the current implementation does not allow users to customise > the volumes used to back SPARK_LOCAL_DIRS to better suit the compute > environment the K8S > It takes time to go through all the properties align them with Spark, but for me that is the correct way to do this at the end of the day. Is the above documented at all for example for the pod template <https://github.com/apache/spark/pull/22146/files>,? Are we missing anything else? Are we going to cover similar cases with more PRs like https://github.com/apache/spark/commit/da6fa3828bb824b65f50122a8a0a0d4741551257 ? Option 3 would take quite some time and would add some code to the backend, not sure of the size of it though, shouldnt be that much. Also I don't think using a pod template is an advanced use of Spark on K8s. Pod templates are used elsewhere by default, for example in kubeflow (for replicas). In addition there are key properties for normal use cases like pod affinity configuration, side car containers etc. These come up often in real deployments. So it depends on who is experiencing what for the UX part. Stavros On Wed, Sep 19, 2018 at 7:41 PM, Erik Erlandson <eerla...@redhat.com> wrote: > I can speak somewhat to the current design. Two of the goals for the > design of this feature are that > (1) its behavior is easy to reason about > (2) its implementation in the back-end is light weight > > Option 1 was chosen partly because it's behavior is relatively simple to > describe to a user: "Your template will be taken as the starting point. > Spark may override a certain small set of fields (documented in a table) > that are necessary for its internal functioning." > > This also keeps the actual back-end implementation relatively light > weight. It can load the template (which also includes syntax validation) > into a pod structure, then modify any fields it needs to (per above). > > > On Wed, Sep 19, 2018 at 9:11 AM, Rob Vesse <rve...@dotnetrdf.org> wrote: > >> Hey all >> >> >> >> For those following the K8S backend you are probably aware of SPARK-24434 >> [1] (and PR 22416 [2]) which proposes a mechanism to allow for advanced pod >> customisation via pod templates. This is motivated by the fact that >> introducing additional Spark configuration properties for each aspect of >> pod specification a user might wish to customise was becoming unwieldy. >> >> >> >> However I am concerned that the current implementation doesn’t go far >> enough and actually limits the utility of the proposed new feature. The >> problem stems from the fact that the implementation simply uses the pod >> template as a base and then Spark attempts to build a pod spec on top of >> that. As the code that does this doesn’t do any kind of validation or >> inspection of the incoming template it is possible to provide a template >> that causes Spark to generate an invalid pod spec ultimately causing the >> job to be rejected by Kubernetes. >> >> >> >> Now clearly Spark code cannot attempt to account for every possible >> customisation that a user may attempt to make via pod templates nor should >> it be responsible for ensuring that the user doesn’t start from an invalid >> template in the first place. However it seems like we could be more >> intelligent in how we build our pod specs to avoid generating invalid specs >> in cases where we have a clear use case for advanced customisation. For >> example the current implementation does not allow users to customise the >> volumes used to back SPARK_LOCAL_DIRS to better suit the compute >> environment the K8S cluster is running on and trying to do so with a pod >> template will result in an invalid spec due to duplicate volumes. >> >> >> >> I think there are a few ways the community could address this: >> >> >> >> 1. Status quo – provide the pod template feature as-is and simply >> tell users that certain customisations are never supported and may result >> in invalid pod specs >> 2. Provide the ability for advanced users to explicitly skip pod spec >> building steps they know interfere with their pod templates via >> configuration properties >> 3. Modify the pod spec building code to be aware of known desirable >> user customisation points and avoid generating invalid specs in those >> cases >> >> >> >> Currently committers seem to be going for Option 1. Personally I would >> like to see the community adopt option 3 but have already received >> considerable pushback when I proposed that in one of my PRs hence the >> suggestion of the compromise option 2. Yes this still has the possibility >> of ending up with invalid specs if users are over-zealous in the spec >> building steps they disable but since this is a power user feature I think >> this would be a risk power users would be willing to assume. If we are >> going to provide features for power users we should avoid unnecessarily >> limiting the utility of those features. >> >> >> >> What do other K8S folks think about this issue? >> >> >> >> Thanks, >> >> >> >> Rob >> >> >> >> [1] https://issues.apache.org/jira/browse/SPARK-24434 >> >> [2] https://github.com/apache/spark/pull/22146 >> >> >> > >
