Andrey Yarovoy created HDDS-17778:
-------------------------------------

             Summary: Replace URIBuilder and http ParseException with JDK 
classes to drop httpclient and httpcore from Ozone client jars
                 Key: HDDS-17778
                 URL: https://issues.apache.org/jira/browse/HDDS-17778
             Project: Apache Ozone
          Issue Type: Improvement
          Components: Ozone Client
            Reporter: Andrey Yarovoy
            Assignee: Andrey Yarovoy


h3. Problem

{\{ozone-common}} and \{{ozone-filesystem-common}} declare \{{httpclient}} 
4.5.14 and \{{httpcore}} 4.4.16 as compile dependencies. The client uses them 
for only two classes, in four places:

||class||used in||for||
|\{{org.apache.http.client.utils.URIBuilder}}|\{{BasicOzoneFileSystem#initialize}},
 \{{BasicRootedOzoneFileSystem#initialize}}|building \{{o3fs://<authority>}} / 
\{{ofs://<authority>}} from scheme and authority|
|\{{org.apache.http.client.utils.URIBuilder}}|\{{OMNodeDetails#getOMDBCheckpointEndpointUrl}}|building
 the OM DB checkpoint URL with two constant boolean parameters|
|\{{org.apache.http.ParseException}}|\{{OFSPath#initOFSPath}}, two 
\{{throw}}s|an unchecked exception for a non-ofs scheme, or when the temp 
bucket name can't be resolved|

The shaded filesystem jar excludes \{{hadoop-common}}, so these declarations 
are the only reason httpclient and httpcore are in it. Their relocated classes 
take *0.89 MB* (compressed) in \{{ozone-filesystem-hadoop3}}:
{noformat}
ozone-filesystem-shaded
\- ozone-filesystem-common:compile
\- httpclient:4.5.14:compile
\- httpcore:4.4.16:compile
{noformat}
That's small, but it's free to remove. The jar is shipped with every job that 
uses \{{-libjars}} or \{{spark.jars}}. It is 155 MB on master, 71 MB with 
HDDS-17772.

h3. Proposed change

# *Filesystem URIs:* replace \{{new 
URIBuilder().setScheme(s).setHost(authority).build()}} with \{{new URI(s, 
authority, null, null, null)}}. Keep \{{URIBuilder}}'s one extra behaviour: it 
brackets an unbracketed IPv6 literal (for \{{ofs://::1/}}, 
\{{URI#getAuthority()}} returns \{{::1}}). Bracket the authority when 
\{{InetAddresses.isInetAddress(authority)}} and it contains \{{:}}. A small 
helper in \{{OzoneFSUtils}} serves both file systems.
# *OM checkpoint URL:* replace it with \{{new URI(scheme, address, path, query, 
null).toURL()}}. The query is 
\{{includeSnapshotData=true&flushBeforeCheckpoint=<flag>}}. Both values are 
constants, so nothing needs encoding.
# *\{{OFSPath}}:* throw \{{IllegalArgumentException}} with the same messages. 
It is unchecked, like \{{org.apache.http.ParseException}}, and nothing in the 
code base catches the http \{{ParseException}}. The three \{{catch 
(ParseException ...)}} in \{{TrashPolicyOzone}}, \{{S3ConditionalRequest}} and 
\{{ReconUtils}} are \{{java.text.ParseException}}.
# *Build:*
#* remove \{{httpclient}} and \{{httpcore}} from 
\{{hadoop-ozone/common/pom.xml}} and \{{hadoop-ozone/ozonefs-common/pom.xml}};
#* add \{{org.apache.httpcomponents}} to the banned dependencies of the client 
modules, next to the RocksDB rule from HDDS-17772.

{\{commons-codec}} also arrives through httpclient, but \{{hdds-common}} 
declares and uses it, so it stays.

h3. Behaviour

Checked \{{URIBuilder}} against \{{java.net.URI}} side by side (the bracketing 
helper of step 1 is not built yet):

||authority||URIBuilder||java.net.URI 5-arg||
|\{{bucket.vol}}, \{{bucket.vol.om1}}, \{{omservice}}|same|same|
|\{{om1:9862}}, \{{bucket.vol.om1:9862}}, \{{10.0.0.1:9874}}|same|same|
|\{{[::1]:9862}}|\{{ofs://[::1]:9862}}|same|
|\{{::1}} (unbracketed)|\{{ofs://[::1]}}|\{{ofs://::1}}, *so bracket it (step 
1)*|
|empty|\{{URISyntaxException}}|\{{URISyntaxException}}|
|\{{https}}, \{{om1:9874}} / \{{[::1]:9875}}, path, two parameters|same|same|

* \{{o3fs}} / \{{ofs}} \{{getUri()}} and the OM checkpoint URL stay 
byte-for-byte the same.
* \{{OFSPath}} throws \{{IllegalArgumentException}} instead of 
\{{org.apache.http.ParseException}}, with the same message.

h3. Compatibility

* No wire, proto, config or on-disk changes.
* Applications that depend on the unshaded \{{ozone-common}} or 
\{{ozone-filesystem-common}} and use httpclient without declaring it lose it as 
a transitive dependency. Inside Ozone, \{{s3gateway}}, \{{cli-shell}}, 
\{{vapor}} and the integration test modules all declare httpclient themselves.

h3. Testing

* *Unit:*
** \{{getUri()}} of both file systems for host, host:port, service id, 
bracketed IPv6 with port, and an unbracketed IPv6 literal;
** \{{getOMDBCheckpointEndpointUrl}} for http and https, with \{{useV2}} and 
\{{flush}} both ways, against the URL that \{{URIBuilder}} builds today;
** \{{OFSPath}} with a non-ofs scheme throws \{{IllegalArgumentException}} with 
the old message.
* *Existing:* \{{TestOzoneFileSystem}}, \{{TestRootedOzoneFileSystem}}, 
\{{TestOFSPath}}, and the OM bootstrap and checkpoint tests.
* *Jar contents:* \{{ozone-filesystem-hadoop3}} has no 
\{{org/apache/ozone/shaded/org/apache/http}} and no 
\{{META-INF/maven/org.apache.httpcomponents}}. Record its size before and after.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to