Andrey Yarovoy created HDDS-17778:
-------------------------------------
Summary: Replace URIBuilder and http ParseException with JDK
classes to drop httpclient and httpcore from Ozone client jars
Key: HDDS-17778
URL: https://issues.apache.org/jira/browse/HDDS-17778
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Client
Reporter: Andrey Yarovoy
Assignee: Andrey Yarovoy
h3. Problem
{\{ozone-common}} and \{{ozone-filesystem-common}} declare \{{httpclient}}
4.5.14 and \{{httpcore}} 4.4.16 as compile dependencies. The client uses them
for only two classes, in four places:
||class||used in||for||
|\{{org.apache.http.client.utils.URIBuilder}}|\{{BasicOzoneFileSystem#initialize}},
\{{BasicRootedOzoneFileSystem#initialize}}|building \{{o3fs://<authority>}} /
\{{ofs://<authority>}} from scheme and authority|
|\{{org.apache.http.client.utils.URIBuilder}}|\{{OMNodeDetails#getOMDBCheckpointEndpointUrl}}|building
the OM DB checkpoint URL with two constant boolean parameters|
|\{{org.apache.http.ParseException}}|\{{OFSPath#initOFSPath}}, two
\{{throw}}s|an unchecked exception for a non-ofs scheme, or when the temp
bucket name can't be resolved|
The shaded filesystem jar excludes \{{hadoop-common}}, so these declarations
are the only reason httpclient and httpcore are in it. Their relocated classes
take *0.89 MB* (compressed) in \{{ozone-filesystem-hadoop3}}:
{noformat}
ozone-filesystem-shaded
\- ozone-filesystem-common:compile
\- httpclient:4.5.14:compile
\- httpcore:4.4.16:compile
{noformat}
That's small, but it's free to remove. The jar is shipped with every job that
uses \{{-libjars}} or \{{spark.jars}}. It is 155 MB on master, 71 MB with
HDDS-17772.
h3. Proposed change
# *Filesystem URIs:* replace \{{new
URIBuilder().setScheme(s).setHost(authority).build()}} with \{{new URI(s,
authority, null, null, null)}}. Keep \{{URIBuilder}}'s one extra behaviour: it
brackets an unbracketed IPv6 literal (for \{{ofs://::1/}},
\{{URI#getAuthority()}} returns \{{::1}}). Bracket the authority when
\{{InetAddresses.isInetAddress(authority)}} and it contains \{{:}}. A small
helper in \{{OzoneFSUtils}} serves both file systems.
# *OM checkpoint URL:* replace it with \{{new URI(scheme, address, path, query,
null).toURL()}}. The query is
\{{includeSnapshotData=true&flushBeforeCheckpoint=<flag>}}. Both values are
constants, so nothing needs encoding.
# *\{{OFSPath}}:* throw \{{IllegalArgumentException}} with the same messages.
It is unchecked, like \{{org.apache.http.ParseException}}, and nothing in the
code base catches the http \{{ParseException}}. The three \{{catch
(ParseException ...)}} in \{{TrashPolicyOzone}}, \{{S3ConditionalRequest}} and
\{{ReconUtils}} are \{{java.text.ParseException}}.
# *Build:*
#* remove \{{httpclient}} and \{{httpcore}} from
\{{hadoop-ozone/common/pom.xml}} and \{{hadoop-ozone/ozonefs-common/pom.xml}};
#* add \{{org.apache.httpcomponents}} to the banned dependencies of the client
modules, next to the RocksDB rule from HDDS-17772.
{\{commons-codec}} also arrives through httpclient, but \{{hdds-common}}
declares and uses it, so it stays.
h3. Behaviour
Checked \{{URIBuilder}} against \{{java.net.URI}} side by side (the bracketing
helper of step 1 is not built yet):
||authority||URIBuilder||java.net.URI 5-arg||
|\{{bucket.vol}}, \{{bucket.vol.om1}}, \{{omservice}}|same|same|
|\{{om1:9862}}, \{{bucket.vol.om1:9862}}, \{{10.0.0.1:9874}}|same|same|
|\{{[::1]:9862}}|\{{ofs://[::1]:9862}}|same|
|\{{::1}} (unbracketed)|\{{ofs://[::1]}}|\{{ofs://::1}}, *so bracket it (step
1)*|
|empty|\{{URISyntaxException}}|\{{URISyntaxException}}|
|\{{https}}, \{{om1:9874}} / \{{[::1]:9875}}, path, two parameters|same|same|
* \{{o3fs}} / \{{ofs}} \{{getUri()}} and the OM checkpoint URL stay
byte-for-byte the same.
* \{{OFSPath}} throws \{{IllegalArgumentException}} instead of
\{{org.apache.http.ParseException}}, with the same message.
h3. Compatibility
* No wire, proto, config or on-disk changes.
* Applications that depend on the unshaded \{{ozone-common}} or
\{{ozone-filesystem-common}} and use httpclient without declaring it lose it as
a transitive dependency. Inside Ozone, \{{s3gateway}}, \{{cli-shell}},
\{{vapor}} and the integration test modules all declare httpclient themselves.
h3. Testing
* *Unit:*
** \{{getUri()}} of both file systems for host, host:port, service id,
bracketed IPv6 with port, and an unbracketed IPv6 literal;
** \{{getOMDBCheckpointEndpointUrl}} for http and https, with \{{useV2}} and
\{{flush}} both ways, against the URL that \{{URIBuilder}} builds today;
** \{{OFSPath}} with a non-ofs scheme throws \{{IllegalArgumentException}} with
the old message.
* *Existing:* \{{TestOzoneFileSystem}}, \{{TestRootedOzoneFileSystem}},
\{{TestOFSPath}}, and the OM bootstrap and checkpoint tests.
* *Jar contents:* \{{ozone-filesystem-hadoop3}} has no
\{{org/apache/ozone/shaded/org/apache/http}} and no
\{{META-INF/maven/org.apache.httpcomponents}}. Record its size before and after.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]