wojiaodoubao commented on code in PR #7294: URL: https://github.com/apache/hadoop/pull/7294#discussion_r1923027537
########## hadoop-cloud-storage-project/hadoop-tos/hadoop-tos-shade/pom.xml: ########## @@ -0,0 +1,132 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd";> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-project</artifactId> + <version>3.5.0-SNAPSHOT</version> + <relativePath>../../../hadoop-project</relativePath> + </parent> + + <artifactId>hadoop-tos-shade</artifactId> + <version>3.5.0-SNAPSHOT</version> + <name>Apache Hadoop Volcano Engine Services Shade Dependencies</name> + <description> + This module contains code to support integration with Volcano Engine TOS. + It also declares the dependencies needed to work with Volcano Engine services. + </description> + <packaging>jar</packaging> + + <properties> + <file.encoding>UTF-8</file.encoding> + <ve-tos-java-sdk.version>2.8.6</ve-tos-java-sdk.version> + <tos.shading.prefix>org.apache.hadoop.fs.tosfs.shaded</tos.shading.prefix> + </properties> + + <dependencyManagement> + <dependencies> + <!-- The ve-tos-java-sdk 2.8.6 depends on okhttp, but okhttp violates dependency convergence. + The com.squareup.okhttp3:okhttp:jar:4.10.0 depends on org.jetbrains.kotlin:kotlin-stdlib-common, + with 2 different versions: 1.5.31 and 1.6.20. Force configure it to 1.6.20. --> + <dependency> + <groupId>org.jetbrains.kotlin</groupId> + <artifactId>kotlin-stdlib-common</artifactId> + <version>1.6.20</version> + </dependency> + </dependencies> + </dependencyManagement> + + <dependencies> + <dependency> + <groupId>com.volcengine</groupId> + <artifactId>ve-tos-java-sdk</artifactId> + <version>${ve-tos-java-sdk.version}</version> + </dependency> + </dependencies> + + <build> + <plugins> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-shade-plugin</artifactId> + <version>${maven-shade-plugin.version}</version> + <executions> + <execution> + <id>shade-tos-sdk</id> + <phase>package</phase> + <goals> + <goal>shade</goal> + </goals> + <configuration> + <shadeSourcesContent>true</shadeSourcesContent> + <createSourcesJar>true</createSourcesJar> + <artifactSet> + <includes> + <include>com.volcengine:*</include> + <include>com.squareup.okhttp3:okhttp</include> + <include>com.squareup.okio:okio-jvm</include> + <include>org.jetbrains.kotlin:kotlin-stdlib*</include> Review Comment: Hi @pan3793 , thanks your comments. The intention of shade is to ensure the tos sdk doesn't conflict with any user's classes, so the hadoop-tos could be simple and safe to use. Here I shaded tos sdk, okhttp, okio and kotlin-stdlib, the final shaded jar is 6,226,348 bytes. So if the downstream project uses okhttp too, the classes will doubled to 12MB+. The default meta space size is 21MB, in the worst case, 12MB+ are all loaded and occupy 57% meta space, which is a problem(may cause long gc). The calculation above is not accurate, since the classes in file is smaller than classes in JVM, and classes won't be loaded until used. But its enough to expose the problem. Your concern is reasonable. We can leave the potential conflicts for downstream projects. They can shade hadoop-tos and it's dependencies if necessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
