Andrey Yarovoy created HDDS-17775:
-------------------------------------
Summary: Remove commons-io, kerby-util and unrelocated macOS
natives from the Ozone filesystem jar
Key: HDDS-17775
URL: https://issues.apache.org/jira/browse/HDDS-17775
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Client
Reporter: Andrey Yarovoy
h3. Problem
Two libraries entered the client dependencies after 2.2.0, each for a single
call. Both are shaded into \{{ozone-filesystem-hadoop3}}, which the 2.2.0 jar
did not include:
||library||added by||client use||compressed in the jar||
|\{{commons-io:commons-io}}|\{{hadoop-hdds/client/pom.xml}}, in the merge of
the HDDS-10685 short-circuit read branch (879707874b)|one
\{{IOUtils.closeQuietly(sock)}} in \{{DomainSocketFactory}}|512 KB, 428 classes|
|\{{org.apache.kerby:kerby-util}}|\{{hadoop-ozone/common/pom.xml}}, by
HDDS-15140, which moved \{{AWSV4AuthValidator}} into \{{ozone-common}}|two
\{{Hex.encode(byte[])}} calls in \{{AWSV4AuthValidator}}|31 KB, 22 classes|
Nothing else in the jar references either library. I scanned the constant pools
of all classes in the shaded jar: each library has exactly one referencing
class.
In addition, two macOS aarch64 netty native libraries ship under their Ratis
names, not relocated:
{noformat}
META-INF/native/liborg_apache_ratis_thirdparty_netty_transport_native_kqueue_aarch_64.jnilib
(82 KB)
META-INF/native/liborg_apache_ratis_thirdparty_netty_resolver_dns_native_macos_aarch_64.jnilib
(59 KB)
{noformat}
{\{ozonefs-shaded/pom.xml}} unpacks the netty natives, renames each to the
Ozone-relocated name with
{\{copy-rename-maven-plugin}}, and excludes the original name in the shade
filter. Both lists have only the x86_64 kqueue
and resolver-dns natives. ratis-thirdparty 1.1.1 added the aarch_64 builds of
both, so they are neither renamed nor
excluded. The relocated classes look for
\{{liborg_apache_ozone_shaded_org_apache_ratis_thirdparty_netty_...}}, so on
Apple-silicon macOS the kqueue transport and the native DNS resolver can't
load. Netty falls back to NIO and the JDK
resolver, which is harmless, but the two files are dead weight.
h3. Proposed fix
* \{{DomainSocketFactory}}: use Ozone's
\{{org.apache.hadoop.hdds.utils.IOUtils.closeQuietly}} (hdds-common), and
remove \{{commons-io}} from \{{hadoop-hdds/client/pom.xml}}. \{{hdds-client}}
has no test uses of commons-io.
* \{{AWSV4AuthValidator}}: use Hadoop's
\{{org.apache.hadoop.util.StringUtils.byteToHexString(byte[])}}. The class
already imports Hadoop \{{StringUtils}} for \{{split}}, so the change needs no
new import or dependency. Remove
{\{kerby-util}} from \{{hadoop-ozone/common/pom.xml}}.
** The output is identical. A check over all 256 byte values and the empty
array gave the same string from kerby
{\{Hex.encode}} and \{{byteToHexString}}, so S3 signature comparison doesn't
change.
** Don't use Ozone's \{{org.apache.hadoop.hdds.StringUtils.bytes2Hex}}. It is a
debug format, uppercase with spaces
(\{{"00 01 0A"}}), and would break every signature check.
** \{{java.util.HexFormat}} is not an option, because \{{ozone-common}} targets
Java 8.
** \{{s3gateway}} keeps its own \{{kerby-util}} dependency
(\{{AWSSignatureProcessor}}, \{{StringToSignProducer}}). It is
server-side and out of scope.
* \{{hadoop-ozone/ozonefs-shaded/pom.xml}}: for
\{{netty_transport_native_kqueue_aarch_64}} and
{\{netty_resolver_dns_native_macos_aarch_64}}, add a
\{{copy-rename-maven-plugin}} \{{fileSet}} to the relocated name and
an \{{<exclude>}} for the Ratis name, following the existing x86_64 entries.
Alternatively, exclude them only, if
Apple-silicon clients don't need the native transport. NIO is the fallback
either way.
* Optional: extend the \{{ban-rocksdb}} enforcer rule from HDDS-17772 (or add a
sibling rule) to ban \{{commons-io}} and
{\{kerby-util}} in the client modules too, so they can't come back unnoticed.
h3. Impact
* \{{ozone-filesystem-hadoop3}} shrinks by about 0.55 MB compressed and 450
entries, plus 36 KB if the natives are only
excluded.
* Two fewer third-party libraries in client classpaths, shaded jars and their
LICENSE/NOTICE.
* commons-io and kerby-util: no behavior change. Natives: with renaming,
Apple-silicon macOS clients can use netty's
kqueue transport and native DNS resolver; with exclusion only, nothing changes,
because they can't load today.
h3. How to check
{noformat}
mvn dependency:tree -pl :ozone-filesystem-shaded
-Dincludes=commons-io,org.apache.kerby # empty after the fix
unzip -l hadoop-ozone/ozonefs-hadoop3/target/ozone-filesystem-hadoop3-*.jar |
grep -cE 'shaded/org/apache/(commons/io|kerby)/' # 0
unzip -l hadoop-ozone/ozonefs-hadoop3/target/ozone-filesystem-hadoop3-*.jar |
grep -c 'native/liborg_apache_ratis_thirdparty' # 0
{noformat}
Tests: \{{TestAWSV4AuthValidator}}, \{{TestS3SecurityUtil}} and
\{{TestS3DerivedKey}} (\{{ozone-manager}}) for the
signature path, and \{{TestXceiverClientManagerSC}} (needs libhadoop) for the
short-circuit path.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]