[
https://issues.apache.org/jira/browse/SPARK-26687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin resolved SPARK-26687.
------------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 23613
[https://github.com/apache/spark/pull/23613]
> Building Spark Images has non-intuitive behaviour with paths to custom
> Dockerfiles
> ----------------------------------------------------------------------------------
>
> Key: SPARK-26687
> URL: https://issues.apache.org/jira/browse/SPARK-26687
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 2.4.0
> Reporter: Rob Vesse
> Assignee: Rob Vesse
> Priority: Major
> Fix For: 3.0.0
>
>
> With the changes from SPARK-26025
> (https://github.com/apache/spark/pull/23019) we use a pared down Docker build
> context which significantly improves build times. However the way this is
> implemented leads to non-intuitive behaviour when supplying custom Docker
> file paths. This is because of the following code snippets:
> {code}
> (cd $(img_ctx_dir base) && docker build $NOCACHEARG "${BUILD_ARGS[@]}" \
> -t $(image_ref spark) \
> -f "$BASEDOCKERFILE" .)
> {code}
> Since the script changes to the temporary build context directory and then
> runs {{docker build}} there any path given for the Docker file is taken as
> relative to the temporary build context directory rather than to the
> directory where the user invoked the script. This produces somewhat
> unhelpful errors e.g.
> {noformat}
> > ./bin/docker-image-tool.sh -r rvesse -t badpath -p
> > resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile
> > build
> Sending build context to Docker daemon 218.4MB
> Step 1/15 : FROM openjdk:8-alpine
> ---> 5801f7d008e5
> Step 2/15 : ARG spark_uid=185
> ---> Using cache
> ---> 5fd63df1ca39
> ...
> Successfully tagged rvesse/spark:badpath
> unable to prepare context: unable to evaluate symlinks in Dockerfile path:
> lstat
> /Users/rvesse/Documents/Work/Code/spark/target/tmp/docker/pyspark/resource-managers:
> no such file or directory
> Failed to build PySpark Docker image, please refer to Docker build output for
> details.
> {noformat}
> Here we can see that the relative path that was valid where the user typed
> the command was not valid inside the build context directory.
> To resolve this we need to ensure that we are resolving relative paths to
> Docker files appropriately.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
