This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d33b3bef86e9 [SPARK-50166][SQL][BUILD] Fix shade and relocation rule
of `sql/core` module
d33b3bef86e9 is described below
commit d33b3bef86e9d701730715b94906b9b10ebb68e9
Author: yangjie01 <[email protected]>
AuthorDate: Fri Nov 1 10:34:41 2024 +0800
[SPARK-50166][SQL][BUILD] Fix shade and relocation rule of `sql/core` module
### What changes were proposed in this pull request?
This pr makes the following changes to the `maven-shade-plugin` rules for
the `sql/core` module:
1. To avoid being influenced by the parent `pom.xml`, use `combine.self =
"override"` in the `<configuration>` of the `maven-shade-plugin` for the
`sql/core` module. Before this configuration was added, the relocation result
was incorrect, and `protobuf-java` was not relocated. We can unzip the
packaging result to confirm this issue.
We can use IntelliJ's "Show Effective POM" feature to view the result of
this parameter, the result is equivalent to the effective POM log with --debug
printing added during the Maven compilation:
**Before**
<img width="828" alt="image"
src="https://github.com/user-attachments/assets/0bce810f-57e9-4a50-9fa2-b6063e040a29";>
We can see that an unexpected
```
<includes>
<include>org.eclipse.jetty.**</include>
</includes>
```
has been added to the relocation rule.
**After**
<img width="787" alt="image"
src="https://github.com/user-attachments/assets/0fab3422-2da7-4b8f-bd7f-9357fcdc39c2";>
We can see that the extra `<includes>` in the relocation rule is no longer
present.
2. Before SPARK-48755 | https://github.com/apache/spark/pull/47133
overwrote the `maven-shade-plugin` rules for `sql/core`, it inherited the rules
from the parent `pom.xml` and shaded `org.spark-project.spark:unused`. This
behavior changed after SPARK-48755, so this pr restores it.
3. The relocation rules for Guava should be retained and follow the
configuration in the parent `pom.xml`, which relocates `com.google.common` to
`${spark.shade.packageName}.guava`. This PR restores this configuration.
4. For `protobuf-java`, which is under the `com.google.protobuf` package,
the already shaded `protobuf-java` in the `core` module can be reused instead
of shading it again in `sql/core` module. Therefore, this pr only configures
the corresponding relocation rule for it: `com.google.protobuf` ->
`${spark.shade.packageName}.spark_core.protobuf`.
5. Regarding the `ServicesResourceTransformer` configuration, it is used to
merge `META-INF/services` resources. This is not needed for Guava and
`protobuf-java`, so this pr removes it.
### Why are the changes needed?
Fix shade and relocation rule of `sql/core` module
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass Github Aciton
- Manually inspect the packaging result: Extract
`spark-sql_2.13-4.0.0-SNAPSHOT.jar` to a separate directory, then execute `grep
"org.sparkproject.guava" -R *` and `grep "org.sparkproject.spark_core.protobuf"
-R *` to confirm the successful relocation.
- Maven test passed: https://github.com/LuciferYang/spark/runs/32278520082
<img width="960" alt="image"
src="https://github.com/user-attachments/assets/5435b2ff-3785-4413-83d9-190c16c6ba75";>
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #48675 from LuciferYang/sql-core-shade.
Lead-authored-by: yangjie01 <[email protected]>
Co-authored-by: YangJie <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
---
sql/core/pom.xml | 19 ++++++++-----------
1 file changed, 8 insertions(+), 11 deletions(-)
diff --git a/sql/core/pom.xml b/sql/core/pom.xml
index 693a968fb498..0a904e3c2746 100644
--- a/sql/core/pom.xml
+++ b/sql/core/pom.xml
@@ -272,27 +272,24 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
- <configuration>
+ <configuration combine.self = "override">
<shadedArtifactAttached>false</shadedArtifactAttached>
<promoteTransitiveDependencies>true</promoteTransitiveDependencies>
<artifactSet>
<includes>
- <include>com.google.protobuf:*</include>
+ <include>org.spark-project.spark:unused</include>
</includes>
</artifactSet>
<relocations>
<relocation>
- <pattern>com.google</pattern>
-
<shadedPattern>${spark.shade.packageName}.com.google</shadedPattern>
- <excludes>
- <!-- Guava is relocated to
${spark.shade.packageName}.connect.guava -->
- <exclude>com.google.common.**</exclude>
- </excludes>
+ <pattern>com.google.common</pattern>
+ <shadedPattern>${spark.shade.packageName}.guava</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>com.google.protobuf</pattern>
+
<shadedPattern>${spark.shade.packageName}.spark_core.protobuf</shadedPattern>
</relocation>
</relocations>
- <transformers>
- <transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
- </transformers>
</configuration>
</plugin>
<!--
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
