This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 140c99c69dd [SPARK-40799][BUILD][CONNECT][FOLLOW-UP] Keep the console
output consistent of lint-scala script as was
140c99c69dd is described below
commit 140c99c69dd2516726552019afb945f9b06a8c1b
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Oct 21 17:38:37 2022 +0900
[SPARK-40799][BUILD][CONNECT][FOLLOW-UP] Keep the console output consistent
of lint-scala script as was
### What changes were proposed in this pull request?
This PR proposes to keep `dev/lint-scala` quiet as was.
### Why are the changes needed?
To remove noisy output from the `dev/lint-scala` script.
**Before**
Success
```
Scalastyle checks passed.
Using `mvn` from path: /.../spark/build/apache-maven-3.8.6/bin/mvn
[INFO] Scanning for projects...
[INFO]
------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO]
------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: x86_64
[INFO] os.detected.version: 10.16
[INFO] os.detected.version.major: 10
[INFO] os.detected.version.minor: 16
[INFO] os.detected.classifier: osx-x86_64
[INFO]
[INFO] ----------------< org.apache.spark:spark-connect_2.12
>-----------------
[INFO] Building Spark Project Connect 3.4.0-SNAPSHOT
[INFO] --------------------------------[ jar
]---------------------------------
[INFO]
[INFO] --- mvn-scalafmt_2.12:1.1.1640084764.9f463a9:format (default-cli)
spark-connect_2.12 ---
[INFO] parsed config (v3.5.9): dev/.scalafmt.conf
[INFO] Scalafmt results: 0 of 11 were unformatted
Details:
Formatted: Connect.scala
Formatted: DataTypeProtoConverter.scala
Formatted: SparkConnectPlanner.scala
Formatted: SparkConnectPlugin.scala
Formatted: SparkConnectCommandPlanner.scala
Formatted: SparkConnectStreamHandler.scala
Formatted: SparkConnectService.scala
Formatted: package.scala
Formatted: SparkConnectProtoSuite.scala
Formatted: SparkConnectPlannerSuite.scala
Formatted: SparkConnectCommandPlannerSuite.scala
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 5.257 s
[INFO] Finished at: 2022-10-21T11:18:19+09:00
[INFO]
------------------------------------------------------------------------
```
Failure
```
Scalastyle checks passed.
Using `mvn` from path:
/Users/hyukjin.kwon/workspace/forked/spark/build/apache-maven-3.8.6/bin/mvn
[INFO] Scanning for projects...
[INFO]
------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO]
------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: x86_64
[INFO] os.detected.version: 10.16
[INFO] os.detected.version.major: 10
[INFO] os.detected.version.minor: 16
[INFO] os.detected.classifier: osx-x86_64
[INFO]
[INFO] ----------------< org.apache.spark:spark-connect_2.12
>-----------------
[INFO] Building Spark Project Connect 3.4.0-SNAPSHOT
[INFO] --------------------------------[ jar
]---------------------------------
[INFO]
[INFO] --- mvn-scalafmt_2.12:1.1.1640084764.9f463a9:format (default-cli)
spark-connect_2.12 ---
[INFO] parsed config (v3.5.9): dev/.scalafmt.conf
[INFO] Scalafmt results: 0 of 11 were unformatted
Details:
Formatted: Connect.scala
Formatted: DataTypeProtoConverter.scala
Formatted: SparkConnectPlanner.scala
Formatted: SparkConnectPlugin.scala
Formatted: SparkConnectCommandPlanner.scala
Formatted: SparkConnectStreamHandler.scala
Formatted: SparkConnectService.scala
Formatted: package.scala
Formatted: SparkConnectProtoSuite.scala
Formatted: SparkConnectPlannerSuite.scala
Formatted: SparkConnectCommandPlannerSuite.scala
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 5.257 s
[INFO] Finished at: 2022-10-21T11:18:19+09:00
[INFO]
------------------------------------------------------------------------
(python3.9) ➜ spark git:(master) ./dev/lint-scala
Scalastyle checks passed.
Using `mvn` from path:
/Users/hyukjin.kwon/workspace/forked/spark/build/apache-maven-3.8.6/bin/mvn
[INFO] Scanning for projects...
[INFO]
------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO]
------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: x86_64
[INFO] os.detected.version: 10.16
[INFO] os.detected.version.major: 10
[INFO] os.detected.version.minor: 16
[INFO] os.detected.classifier: osx-x86_64
[INFO]
[INFO] ----------------< org.apache.spark:spark-connect_2.12
>-----------------
[INFO] Building Spark Project Connect 3.4.0-SNAPSHOT
[INFO] --------------------------------[ jar
]---------------------------------
[INFO]
[INFO] --- mvn-scalafmt_2.12:1.1.1640084764.9f463a9:format (default-cli)
spark-connect_2.12 ---
[INFO] parsed config (v3.5.9): dev/.scalafmt.conf
[ERROR] unformatted file at:
/Users/hyukjin.kwon/workspace/forked/spark/connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
[INFO] Scalafmt results: 1 of 11 were unformatted
Details:
Formatted: Connect.scala
Formatted: DataTypeProtoConverter.scala
Requires formatting: SparkConnectPlanner.scala
Formatted: SparkConnectPlugin.scala
Formatted: SparkConnectCommandPlanner.scala
Formatted: SparkConnectStreamHandler.scala
Formatted: SparkConnectService.scala
Formatted: package.scala
Formatted: SparkConnectProtoSuite.scala
Formatted: SparkConnectPlannerSuite.scala
Formatted: SparkConnectCommandPlannerSuite.scala
[ERROR]
org.apache.maven.plugin.MojoExecutionException: Scalafmt: Unformatted files
found
at org.antipathy.mvn_scalafmt.FormatMojo.execute (FormatMojo.java:91)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo
(DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2
(MojoExecutor.java:370)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute
(MojoExecutor.java:351)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:171)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:163)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
(LifecycleModuleBuilder.java:117)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
(LifecycleModuleBuilder.java:81)
at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
(SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute
(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced
(Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch
(Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode
(Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main
(Launcher.java:347)
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 5.223 s
[INFO] Finished at: 2022-10-21T11:19:58+09:00
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.antipathy:mvn-scalafmt_2.12:1.1.1640084764.9f463a9:format (default-cli) on
project spark-connect_2.12: Error formatting Scala files: Scalafmt: Unformatted
files found -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
The scalafmt check failed on connector/connect.
Before submitting your change, please make sure to format your code using
the following command:
./build/mvn -Pscala-2.12 scalafmt:format -Dscalafmt.skip=fase
-Dscalafmt.validateOnly=false -Dscalafmt.changedOnly=false -pl connector/connect
```
**After**
Success
```
Scalastyle checks passed.
Scalafmt checks passed.
```
Failure
```
Scalastyle checks passed.
The scalafmt check failed on connector/connect at following occurrences:
Requires formatting: SparkConnectPlanner.scala
Before submitting your change, please make sure to format your code using
the following command:
./build/mvn -Pscala-2.12 scalafmt:format -Dscalafmt.skip=fase
-Dscalafmt.validateOnly=false -Dscalafmt.changedOnly=false -pl connector/connect
```
In this way, this is consistent before
https://github.com/apache/spark/pull/38258.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
Manually tested as described above.
Closes #38326 from HyukjinKwon/SPARK-40799-followup.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
dev/lint-scala | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/dev/lint-scala b/dev/lint-scala
index ad2be152cfa..ea3b98464b2 100755
--- a/dev/lint-scala
+++ b/dev/lint-scala
@@ -23,10 +23,21 @@ SPARK_ROOT_DIR="$(dirname $SCRIPT_DIR)"
"$SCRIPT_DIR/scalastyle" "$1"
# For Spark Connect, we actively enforce scalafmt and check that the produced
diff is empty.
-./build/mvn -Pscala-2.12 scalafmt:format -Dscalafmt.skip=false
-Dscalafmt.validateOnly=true -Dscalafmt.changedOnly=false -pl connector/connect
-if [[ $? -ne 0 ]]; then
- echo "The scalafmt check failed on connector/connect."
+ERRORS=$(./build/mvn \
+ -Pscala-2.12 \
+ scalafmt:format \
+ -Dscalafmt.skip=false \
+ -Dscalafmt.validateOnly=true \
+ -Dscalafmt.changedOnly=false \
+ -pl connector/connect \
+ 2>&1 | grep -e "^Requires formatting" \
+)
+
+if test ! -z "$ERRORS"; then
+ echo -e "The scalafmt check failed on connector/connect at following
occurrences:\n\n$ERRORS\n"
echo "Before submitting your change, please make sure to format your code
using the following command:"
echo "./build/mvn -Pscala-2.12 scalafmt:format -Dscalafmt.skip=fase
-Dscalafmt.validateOnly=false -Dscalafmt.changedOnly=false -pl
connector/connect"
exit 1
+else
+ echo -e "Scalafmt checks passed."
fi
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
