[ 
https://issues.apache.org/jira/browse/HDDS-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Doroszlai updated HDDS-17775:
------------------------------------
    Fix Version/s: 2.3.0
       Resolution: Fixed
           Status: Resolved  (was: Patch Available)

> Remove commons-io, kerby-util and unrelocated macOS natives from the Ozone 
> filesystem jar
> -----------------------------------------------------------------------------------------
>
>                 Key: HDDS-17775
>                 URL: https://issues.apache.org/jira/browse/HDDS-17775
>             Project: Apache Ozone
>          Issue Type: Bug
>          Components: Ozone Client
>            Reporter: Andrey Yarovoy
>            Assignee: Andrey Yarovoy
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 2.3.0
>
>
> h3. Problem
> Two libraries entered the client dependencies after 2.2.0, each for a single 
> call. Both are shaded into \{{ozone-filesystem-hadoop3}}, which the 2.2.0 jar 
> did not include:
> ||library||added by||client use||compressed in the jar||
> |\{{commons-io:commons-io}}|\{{hadoop-hdds/client/pom.xml}}, in the merge of 
> the HDDS-10685 short-circuit read branch (879707874b)|one 
> \{{IOUtils.closeQuietly(sock)}} in \{{DomainSocketFactory}}|512 KB, 428 
> classes|
> |\{{org.apache.kerby:kerby-util}}|\{{hadoop-ozone/common/pom.xml}}, by 
> HDDS-15140, which moved \{{AWSV4AuthValidator}} into \{{ozone-common}}|two 
> \{{Hex.encode(byte[])}} calls in \{{AWSV4AuthValidator}}|31 KB, 22 classes|
> Nothing else in the jar references either library. I scanned the constant 
> pools of all classes in the shaded jar: each library has exactly one 
> referencing class.
> In addition, two macOS aarch64 netty native libraries ship under their Ratis 
> names, not relocated:
> {noformat}
> META-INF/native/liborg_apache_ratis_thirdparty_netty_transport_native_kqueue_aarch_64.jnilib
>  (82 KB)
> META-INF/native/liborg_apache_ratis_thirdparty_netty_resolver_dns_native_macos_aarch_64.jnilib
>  (59 KB)
> {noformat}
> {\{ozonefs-shaded/pom.xml}} unpacks the netty natives, renames each to the 
> Ozone-relocated name with
> {\{copy-rename-maven-plugin}}, and excludes the original name in the shade 
> filter. Both lists have only the x86_64 kqueue
> and resolver-dns natives. ratis-thirdparty 1.1.1 added the aarch_64 builds of 
> both, so they are neither renamed nor
> excluded. The relocated classes look for 
> \{{liborg_apache_ozone_shaded_org_apache_ratis_thirdparty_netty_...}}, so on
> Apple-silicon macOS the kqueue transport and the native DNS resolver can't 
> load. Netty falls back to NIO and the JDK
> resolver, which is harmless, but the two files are dead weight.
> h3. Proposed fix
> * \{{DomainSocketFactory}}: use Ozone's 
> \{{org.apache.hadoop.hdds.utils.IOUtils.closeQuietly}} (hdds-common), and
> remove \{{commons-io}} from \{{hadoop-hdds/client/pom.xml}}. \{{hdds-client}} 
> has no test uses of commons-io.
> * \{{AWSV4AuthValidator}}: use Hadoop's 
> \{{org.apache.hadoop.util.StringUtils.byteToHexString(byte[])}}. The class
> already imports Hadoop \{{StringUtils}} for \{{split}}, so the change needs 
> no new import or dependency. Remove
> {\{kerby-util}} from \{{hadoop-ozone/common/pom.xml}}.
> ** The output is identical. A check over all 256 byte values and the empty 
> array gave the same string from kerby
> {\{Hex.encode}} and \{{byteToHexString}}, so S3 signature comparison doesn't 
> change.
> ** Don't use Ozone's \{{org.apache.hadoop.hdds.StringUtils.bytes2Hex}}. It is 
> a debug format, uppercase with spaces
> (\{{"00 01 0A"}}), and would break every signature check.
> ** \{{java.util.HexFormat}} is not an option, because \{{ozone-common}} 
> targets Java 8.
> ** \{{s3gateway}} keeps its own \{{kerby-util}} dependency 
> (\{{AWSSignatureProcessor}}, \{{StringToSignProducer}}). It is
> server-side and out of scope.
> * \{{hadoop-ozone/ozonefs-shaded/pom.xml}}: for 
> \{{netty_transport_native_kqueue_aarch_64}} and
> {\{netty_resolver_dns_native_macos_aarch_64}}, add a 
> \{{copy-rename-maven-plugin}} \{{fileSet}} to the relocated name and
> an \{{<exclude>}} for the Ratis name, following the existing x86_64 entries. 
> Alternatively, exclude them only, if
> Apple-silicon clients don't need the native transport. NIO is the fallback 
> either way.
> * Optional: extend the \{{ban-rocksdb}} enforcer rule from HDDS-17772 (or add 
> a sibling rule) to ban \{{commons-io}} and
> {\{kerby-util}} in the client modules too, so they can't come back unnoticed.
> h3. Impact
> * \{{ozone-filesystem-hadoop3}} shrinks by about 0.55 MB compressed and 450 
> entries, plus 36 KB if the natives are only
> excluded.
> * Two fewer third-party libraries in client classpaths, shaded jars and their 
> LICENSE/NOTICE.
> * commons-io and kerby-util: no behavior change. Natives: with renaming, 
> Apple-silicon macOS clients can use netty's
> kqueue transport and native DNS resolver; with exclusion only, nothing 
> changes, because they can't load today.
> h3. How to check
> {noformat}
> mvn dependency:tree -pl :ozone-filesystem-shaded 
> -Dincludes=commons-io,org.apache.kerby # empty after the fix
> unzip -l hadoop-ozone/ozonefs-hadoop3/target/ozone-filesystem-hadoop3-*.jar | 
> grep -cE 'shaded/org/apache/(commons/io|kerby)/' # 0
> unzip -l hadoop-ozone/ozonefs-hadoop3/target/ozone-filesystem-hadoop3-*.jar | 
> grep -c 'native/liborg_apache_ratis_thirdparty' # 0
> {noformat}
> Tests: \{{TestAWSV4AuthValidator}}, \{{TestS3SecurityUtil}} and 
> \{{TestS3DerivedKey}} (\{{ozone-manager}}) for the
> signature path, and \{{TestXceiverClientManagerSC}} (needs libhadoop) for the 
> short-circuit path.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to