[ 
https://issues.apache.org/jira/browse/HDDS-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HDDS-17778:
----------------------------------
    Labels: pull-request-available  (was: )

> Replace URIBuilder and http ParseException with JDK classes to drop 
> httpclient and httpcore from Ozone client jars
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17778
>                 URL: https://issues.apache.org/jira/browse/HDDS-17778
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: Ozone Client
>            Reporter: Andrey Yarovoy
>            Assignee: Andrey Yarovoy
>            Priority: Major
>              Labels: pull-request-available
>
> h3. Problem
> {{ozone-common}} and {{ozone-filesystem-common}} declare {{httpclient}} 
> 4.5.14 and {{httpcore}} 4.4.16 as compile dependencies. The client uses them 
> for only two classes, in four places:
> ||class||used in||for||
> |{{org.apache.http.client.utils.URIBuilder}}|{{{}BasicOzoneFileSystem#initialize{}}},
>  {{BasicRootedOzoneFileSystem#initialize}}|building {{o3fs://<authority>}} / 
> {{ofs://<authority>}} from scheme and authority|
> |{{org.apache.http.client.utils.URIBuilder}}|{{OMNodeDetails#getOMDBCheckpointEndpointUrl}}|building
>  the OM DB checkpoint URL with two constant boolean parameters|
> |{{org.apache.http.ParseException}}|{{{}OFSPath#initOFSPath{}}}, two 
> {{throw}} statements|an unchecked exception for a non-ofs scheme, or when the 
> temp bucket name can't be resolved|
>  
>  
>  
> The shaded filesystem jar excludes {{{}hadoop-common{}}}, so these 
> declarations are the only reason httpclient and httpcore are in it. Their 
> relocated classes take *0.89 MB* (compressed) in 
> {{{}ozone-filesystem-hadoop3{}}}:
> {noformat}
> ozone-filesystem-shaded
> \- ozone-filesystem-common:compile
> \- httpclient:4.5.14:compile
> \- httpcore:4.4.16:compile
> {noformat}
> That's small, but it's free to remove. The jar is shipped with every job that 
> uses {{-libjars}} or {{{}spark.jars{}}}. It is 155 MB on master, 71 MB with 
> HDDS-17772.
> h3. Proposed change
>  # *Filesystem URIs:* replace {{new 
> URIBuilder().setScheme(s).setHost(authority).build()}} with {{{}new URI(s, 
> authority, null, null, null){}}}. Keep {{{}URIBuilder{}}}'s one extra 
> behaviour: it brackets an unbracketed IPv6 literal (for {{{}ofs://::1/{}}}, 
> {{URI#getAuthority()}} returns {{{}::1{}}}). Bracket the authority when 
> {{InetAddresses.isInetAddress(authority)}} and it contains {{{}:{}}}. A small 
> helper in {{OzoneFSUtils}} serves both file systems.
>  # *OM checkpoint URL:* replace it with {{{}new URI(scheme, address, path, 
> query, null).toURL(){}}}. The query is 
> {{{}includeSnapshotData=true&flushBeforeCheckpoint=<flag>{}}}. Both values 
> are constants, so nothing needs encoding.
>  # *{{{}OFSPath{}}}:* throw {{IllegalArgumentException}} with the same 
> messages. It is unchecked, like {{{}org.apache.http.ParseException{}}}, and 
> nothing in the code base catches the http {{{}ParseException{}}}. The three 
> {{catch (ParseException ...)}} in {{{}TrashPolicyOzone{}}}, 
> {{S3ConditionalRequest}} and {{ReconUtils}} are 
> {{{}java.text.ParseException{}}}.
>  # *Build:*
>  ** remove {{httpclient}} and {{httpcore}} from 
> {{hadoop-ozone/common/pom.xml}} and 
> {{{}hadoop-ozone/ozonefs-common/pom.xml{}}};
>  ** add {{org.apache.httpcomponents}} to the banned dependencies of the 
> client modules, next to the RocksDB rule from HDDS-17772.
> {{commons-codec}} also arrives through httpclient, but {{hdds-common}} 
> declares and uses it, so it stays.
> h3. Behaviour
> Checked {{URIBuilder}} against {{java.net.URI}} side by side (the bracketing 
> helper of step 1 is not built yet):
>  
> ||authority||URIBuilder||java.net.URI 5-arg||
> |\{{bucket.vol}}, \{{bucket.vol.om1}}, \{{omservice}} |same|same|
> |\{{om1:9862}}, \{{bucket.vol.om1:9862}}, \{{10.0.0.1:9874}} |same|same|
> |\{{[::1]:9862}} |\{{ofs://[::1]:9862}} |same|
> |\{{::1}} (unbracketed)|\{{ofs://[::1]}} |\{{ofs://::1}}, but unreachable: 
> both file systems reject it first (2.3.0-SNAPSHOT, all four forms 
> \{{ofs://::1/}}, \{{ofs://::1:9862/}}, \{{o3fs://bucket.vol.::1/}}, 
> \{{o3fs://bucket.vol.::1:9862/}})|
> |empty|\{{URISyntaxException}} |\{{URISyntaxException}} |
> |\{{https}}, \{{om1:9874}} / \{{[::1]:9875}}, path, two parameters|same|same|
> {{o3fs}} / {{ofs}} {{getUri()}} and the OM checkpoint URL stay byte-for-byte 
> the same.
> {{OFSPath}} throws {{IllegalArgumentException}} instead of 
> {{{}org.apache.http.ParseException{}}}, with the same message.
> h3. Compatibility
>  * No wire, proto, config or on-disk changes.
>  * Applications that depend on the unshaded {{ozone-common}} or 
> {{ozone-filesystem-common}} and use httpclient without declaring it lose it 
> as a transitive dependency. Inside Ozone, {{{}s3gateway{}}}, 
> {{{}cli-shell{}}}, {{vapor}} and the integration test modules all declare 
> httpclient themselves.
> h3. Testing
>  * *Unit:*
>  ** {{getUri()}} of both file systems for host, host:port, service id, 
> bracketed IPv6 with port, and an unbracketed IPv6 literal;
>  ** {{getOMDBCheckpointEndpointUrl}} for http and https, with {{useV2}} and 
> {{flush}} both ways, against the URL that {{URIBuilder}} builds today;
>  ** {{OFSPath}} with a non-ofs scheme throws {{IllegalArgumentException}} 
> with the old message.
>  * *Existing:* {{{}TestOzoneFileSystem{}}}, 
> {{{}TestRootedOzoneFileSystem{}}}, {{{}TestOFSPath{}}}, and the OM bootstrap 
> and checkpoint tests.
>  * *Jar contents:* {{ozone-filesystem-hadoop3}} has no 
> {{org/apache/ozone/shaded/org/apache/http}} and no 
> {{{}META-INF/maven/org.apache.httpcomponents{}}}. Record its size before and 
> after.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to