philwalk opened a new pull request, #38228:
URL: https://github.com/apache/spark/pull/38228
### Build Problem
This fixes two problems that affect development in a Windows shell
environment, such as `cygwin` or `msys2`.
Running `./build/sbt packageBin` from A Windows cygwin `bash` session fails.
This occurs if `WSL` is installed, because `project\SparkBuild.scala`
creates a `bash` process, but `WSL bash` is called, even though `cygwin bash`
appears earlier in the `PATH`. In addition, file path arguments to bash
contain backslashes. The fix is to insure that the correct `bash` is called,
and that arguments passed to `bash` are passed with slashes rather than
**slashes.**
### The build error message:
```bash
./build.sbt packageBin
```
<pre>
[info] compiling 9 Java sources to
C:\Users\philwalk\workspace\spark\common\sketch\target\scala-2.12\classes ...
/bin/bash: C:Usersphilwalkworkspacesparkcore/../build/spark-build-info: No
such file or directory
[info] compiling 1 Scala source to
C:\Users\philwalk\workspace\spark\tools\target\scala-2.12\classes ...
[info] compiling 5 Scala sources to
C:\Users\philwalk\workspace\spark\mllib-local\target\scala-2.12\classes ...
[info] Compiling 5 protobuf files to
C:\Users\philwalk\workspace\spark\connector\connect\target\scala-2.12\src_managed\main
[error] stack trace is suppressed; run last core / Compile /
managedResources for the full output
[error] (core / Compile / managedResources) Nonzero exit value: 127
[error] Total time: 42 s, completed Oct 8, 2022, 4:49:12 PM
sbt:spark-parent>
sbt:spark-parent> last core /Compile /managedResources
last core /Compile /managedResources
[error] java.lang.RuntimeException: Nonzero exit value: 127
[error] at scala.sys.package$.error(package.scala:30)
[error] at
scala.sys.process.ProcessBuilderImpl$AbstractBuilder.slurp(ProcessBuilderImpl.scala:138)
[error] at
scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang$bang(ProcessBuilderImpl.scala:108)
[error] at Core$.$anonfun$settings$4(SparkBuild.scala:604)
[error] at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] at
sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error] at sbt.std.Transform$$anon$4.work(Transform.scala:68)
[error] at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
[error] at
sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
[error] at sbt.Execute.work(Execute.scala:291)
[error] at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
[error] at
sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error] at
sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
[error] at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[error] at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[error] at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[error] at java.base/java.lang.Thread.run(Thread.java:834)
[error] (core / Compile / managedResources) Nonzero exit value: 127
</pre>
### bash scripts fail when run from `cygwin` or `msys2`
The other problem fixed by the PR is to address problems preventing the
`bash` scripts (`spark-shell`, `spark-submit`, etc.) from being used in Windows
`SHELL` environments. The problem is that the bash version of `spark-class`
fails in a Windows shell environment, the result of
`launcher/src/main/java/org/apache/spark/launcher/Main.java` not following the
convention expected by `spark-class`, and also appending CR to line endings.
The resulting error message not helpful.
There are two parts to this fix:
1. modify `Main.java` to treat a `SHELL` session on Windows as a `bash`
session
2. remove the appended CR character when parsing the output produced by
`Main.java`
### Does this PR introduce _any_ user-facing change?
These changes should NOT affect anyone who is not trying build or run bash
scripts from a Windows SHELL environment.
### How was this patch tested?
Manual tests were performed to verify both changes.
### related JIRA issues
The following 2 JIRA issue were created. Both are fixed by this PR. They
are both linked to this PR.
- Bug SPARK-40739 "sbt packageBin" fails in cygwin or other windows bash
session
- Bug SPARK-40738 spark-shell fails with "bad array"
-
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
