LuciferYang commented on code in PR #47402:
URL: https://github.com/apache/spark/pull/47402#discussion_r1709039887
##########
assembly/pom.xml:
##########
@@ -159,6 +159,78 @@
</target>
</configuration>
</plugin>
+ <plugin>
+ <!--
+ Here we download ammonite dependency required for Spark Connect REPL
and copy
+ Spark Connect client to target's jars/connect-repl directory. Both
jars will
+ only be loaded when we run Spark Connect shell, see also
+ AbstractCommandBuilder.buildClassPath and SPARK-48936.
+ -->
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>exec-maven-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>get-ammonite-jar</id>
+ <phase>package</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+ <executable>${basedir}/../build/mvn</executable>
Review Comment:
When I execute `build/mvn clean package -DskipTests -pl assembly -am`, only
24 modules are built. Since `assembly` does not depend on the
`connect-client-jvm` module, the `connect-client-jvm` module is not built,
leading to a similar error mentioned by @pan3793 earlier. Maybe this is the
same issue?
```
[INFO] --- exec:3.2.0:exec (get-connect-client-jar) @ spark-assembly_2.13 ---
cp:
/Users/yangjie01/SourceCode/git/spark-mine-13/assembly/../connector/connect/client/jvm/target/spark-connect-client-jvm_2.13-4.0.0-SNAPSHOT.jar:
No such file or directory
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1
(Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal
(DefaultExecutor.java:355)
at org.apache.commons.exec.DefaultExecutor.execute
(DefaultExecutor.java:253)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:884)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:844)
at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:450)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo
(DefaultBuildPluginManager.java:126)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2
(MojoExecutor.java:328)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute
(MojoExecutor.java:316)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:212)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:174)
at org.apache.maven.lifecycle.internal.MojoExecutor.access$000
(MojoExecutor.java:75)
at org.apache.maven.lifecycle.internal.MojoExecutor$1.run
(MojoExecutor.java:162)
at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute
(DefaultMojosExecutionStrategy.java:39)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:159)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
(LifecycleModuleBuilder.java:105)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
(LifecycleModuleBuilder.java:73)
at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
(SingleThreadedBuilder.java:53)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute
(LifecycleStarter.java:118)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:903)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:280)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:203)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:77)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:568)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced
(Launcher.java:255)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch
(Launcher.java:201)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode
(Launcher.java:361)
at org.codehaus.plexus.classworlds.launcher.Launcher.main
(Launcher.java:314)
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 4.0.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 2.381
s]
[INFO] Spark Project Tags ................................. SUCCESS [ 4.217
s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 6.052
s]
[INFO] Spark Project Common Utils ......................... SUCCESS [ 15.559
s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 5.974
s]
[INFO] Spark Project Networking ........................... SUCCESS [ 7.780
s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 6.530
s]
[INFO] Spark Project Variant .............................. SUCCESS [ 2.353
s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 7.575
s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 4.775
s]
[INFO] Spark Project Core ................................. SUCCESS [01:08
min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 14.716
s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 16.256
s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 23.458
s]
[INFO] Spark Project SQL API .............................. SUCCESS [ 12.449
s]
[INFO] Spark Project Catalyst ............................. SUCCESS [ 58.710
s]
[INFO] Spark Project SQL .................................. SUCCESS [01:39
min]
[INFO] Spark Project ML Library ........................... SUCCESS [01:05
min]
[INFO] Spark Project Connect Common ....................... SUCCESS [ 21.806
s]
[INFO] Spark Avro ......................................... SUCCESS [ 18.595
s]
[INFO] Spark Protobuf ..................................... SUCCESS [ 18.957
s]
[INFO] Spark Project REPL ................................. SUCCESS [ 11.450
s]
[INFO] Spark Project Connect Server ....................... SUCCESS [ 29.381
s]
[INFO] Spark Project Assembly ............................. FAILURE [ 27.147
s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
