This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new bd37b10c756 [SPARK-41377][BUILD] Fix spark-version-info.properties not
found on Windows
bd37b10c756 is described below
commit bd37b10c7568b0fe07c98da5a532c5ff84d702f6
Author: Gautham Banasandra <[email protected]>
AuthorDate: Fri Dec 9 02:06:12 2022 -0800
[SPARK-41377][BUILD] Fix spark-version-info.properties not found on Windows
### What changes were proposed in this pull request?
This PR enhances the Maven build configuration to automatically detect and
switch between using Powershell for Windows and Bash for non-Windows OS to
generate `spark-version-info.properties` file.
### Why are the changes needed?
While building Spark, the `spark-version-info.properties` file [is
generated using
bash](https://github.com/apache/spark/blob/d62c18b7497997188ec587e1eb62e75c979c1c93/core/pom.xml#L560-L564).
In Windows environment, if Windows Subsystem for Linux (WSL) is installed, it
somehow overrides the other bash executables in the PATH, as noted in
SPARK-40739. The bash in WSL has a different mounting configuration and thus,
[the target location specified for spark-version-info.properties](https [...]
This PR fixes the issue by directing the build system to use the right
shell according to the platform.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
I tested this by building on a Windows 10 PC.
```psh
mvn -Pyarn '-Dhadoop.version=3.3.0' -DskipTests clean package
```
Once the build finished, I verified that `spark-version-info.properties`
file was included in the spark-core jar.
I also ran the SparkPi application and verified that it ran successfully
without any errors.
Closes #38903 from GauthamBanasandra/spark-version-info-ps.
Authored-by: Gautham Banasandra <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
appveyor.yml | 1 +
build/spark-build-info.ps1 | 46 ++++++++++++++++++++++++++++++++++++++++++++++
core/pom.xml | 34 ++++++++++++++++++++++++++++++++--
3 files changed, 79 insertions(+), 2 deletions(-)
diff --git a/appveyor.yml b/appveyor.yml
index 1a2aef0d3b8..fdb247d5d43 100644
--- a/appveyor.yml
+++ b/appveyor.yml
@@ -28,6 +28,7 @@ only_commits:
files:
- appveyor.yml
- dev/appveyor-install-dependencies.ps1
+ - build/spark-build-info.ps1
- R/
- sql/core/src/main/scala/org/apache/spark/sql/api/r/
- core/src/main/scala/org/apache/spark/api/r/
diff --git a/build/spark-build-info.ps1 b/build/spark-build-info.ps1
new file mode 100644
index 00000000000..43db8823340
--- /dev/null
+++ b/build/spark-build-info.ps1
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This script generates the build info for spark and places it into the
spark-version-info.properties file.
+# Arguments:
+# ResourceDir - The target directory where properties file would be created.
[./core/target/extra-resources]
+# SparkVersion - The current version of spark
+
+param(
+ # The resource directory.
+ [Parameter(Position = 0)]
+ [String]
+ $ResourceDir,
+
+ # The Spark version.
+ [Parameter(Position = 1)]
+ [String]
+ $SparkVersion
+)
+
+$null = New-Item -Type Directory -Force $ResourceDir
+$SparkBuildInfoPath = $ResourceDir.TrimEnd('\').TrimEnd('/') +
'\spark-version-info.properties'
+
+$SparkBuildInfoContent =
+"version=$SparkVersion
+user=$($Env:USERNAME)
+revision=$(git rev-parse HEAD)
+branch=$(git rev-parse --abbrev-ref HEAD)
+date=$([DateTime]::UtcNow | Get-Date -UFormat +%Y-%m-%dT%H:%M:%SZ)
+url=$(git config --get remote.origin.url)"
+
+Set-Content -Path $SparkBuildInfoPath -Value $SparkBuildInfoContent
diff --git a/core/pom.xml b/core/pom.xml
index a9b40acf5a3..470140fd883 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -558,12 +558,42 @@
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
+ <id>choose-shell-and-script</id>
+ <phase>validate</phase>
+ <goals>
+ <goal>run</goal>
+ </goals>
+ <configuration>
+ <exportAntProperties>true</exportAntProperties>
+ <target>
+ <condition property="shell" value="powershell.exe" else="bash">
+ <and>
+ <os family="windows"/>
+ </and>
+ </condition>
+ <condition property="spark-build-info-script"
value="spark-build-info.ps1"
+ else="spark-build-info">
+ <and>
+ <os family="windows"/>
+ </and>
+ </condition>
+ <echo>Shell to use for generating
spark-version-info.properties file =
+ ${shell}
+ </echo>
+ <echo>Script to use for generating
spark-version-info.properties file =
+ ${spark-build-info-script}
+ </echo>
+ </target>
+ </configuration>
+ </execution>
+ <execution>
+ <id>generate-spark-build-info</id>
<phase>generate-resources</phase>
<configuration>
<!-- Execute the shell script to generate the spark build
information. -->
<target>
- <exec executable="bash">
- <arg value="${project.basedir}/../build/spark-build-info"/>
+ <exec executable="${shell}">
+ <arg
value="${project.basedir}/../build/${spark-build-info-script}"/>
<arg value="${project.build.directory}/extra-resources"/>
<arg value="${project.version}"/>
</exec>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
