[
https://issues.apache.org/jira/browse/HADOOP-15965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16710014#comment-16710014
]
Vishwajeet Dusane edited comment on HADOOP-15965 at 12/5/18 12:37 PM:
----------------------------------------------------------------------
Thanks [[email protected]].
{quote} * make sure it works when wildfly isn't on the CP, and doesn't bother
warning people at info/warn{quote} * Wildfly dependency is shaded with Adls
Java SDK hence no dependency package is needed. Verified that even if native
libraries are failed to load, Java SDK uses JSE instead. However, not
recommended since would hit hard on performance.
* For info/warn - You might be referring to HADOOP-15851? In that case, Java
SDK 2.3.2 does not incorporate this change. Java SDK 2.3.3 would have the
change, timeline yet to be decided. Would it be fine to bypass this check for
this Jira since not a functional issue? We have to unblock priority customer
hence request to relax on this logline.
{quote} * can you make sure the SDK checks content tyoe of Auth & raises
meaningful error if != json{quote} * This issue is addressed in the [Java SDK
2.3.2
version|[https://github.com/Azure/azure-data-lake-store-java/blob/dafdcdef608f13e951b0336a44307a7b5f1e894b/src/main/java/com/microsoft/azure/datalake/store/oauth2/AzureADAuthenticator.java#L266].]
{quote} * review all transitive dependencies: is it going to force in new
versions of anything?{quote} * No other dependency version changes.
{quote} * and of course, usual "test everything process" applies{quote}
{code:java}
[INFO] Tests run: 470, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
134.406 s - in org.apache.hadoop.fs.adl.live.TestAdlSupportedCharsetInPath
[INFO] Running org.apache.hadoop.fs.adl.live.TestMetadata
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.372 s
- in org.apache.hadoop.fs.adl.live.TestMetadata
[INFO] Running org.apache.hadoop.fs.adl.TestAzureADTokenProvider
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.952 s
- in org.apache.hadoop.fs.adl.TestAzureADTokenProvider
[INFO] Running org.apache.hadoop.fs.adl.TestRelativePathFormation
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.1 s -
in org.apache.hadoop.fs.adl.TestRelativePathFormation
[INFO] Running org.apache.hadoop.fs.adl.TestValidateConfiguration
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.154 s
- in org.apache.hadoop.fs.adl.TestValidateConfiguration
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 869, Failures: 0, Errors: 0, Skipped: 4
{code}
was (Author: vishwajeet.dusane):
Thanks [[email protected]].
{quote} * make sure it works when wildfly isn't on the CP, and doesn't bother
warning people at info/warn{quote} * Wildfly dependency is shaded with Adls
Java SDK hence no dependency package is needed. Verified that even if native
libraries are failed to load, Java SDK uses JSE instead. However, not
recommended since would hit hard on performance.
* For info/warn - You might be referring to HADOOP-15851? In that case, Java
SDK 2.3.2 does not incorporate this change. Java SDK 2.3.3 would have the
change, timeline yet to be decided. Would it be fine to bypass this check for
this Jira since not a functional issue? We have to unblock priority customer
hence request to relax on this logline.
{quote} * can you make sure the SDK checks content tyoe of Auth & raises
meaningful error if != json{quote} * This issue is addressed in the [Java SDK
2.3.2
version|[https://github.com/Azure/azure-data-lake-store-java/blob/dafdcdef608f13e951b0336a44307a7b5f1e894b/src/main/java/com/microsoft/azure/datalake/store/oauth2/AzureADAuthenticator.java#L266].]
{quote} * review all transitive dependencies: is it going to force in new
versions of anything?{quote} * No other dependency version changes.
{quote} * and of course, usual "test everything process" applies{quote}
{code:java}
[INFO] Tests run: 470, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
134.406 s - in org.apache.hadoop.fs.adl.live.TestAdlSupportedCharsetInPath
[INFO] Running org.apache.hadoop.fs.adl.live.TestMetadata
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.372 s
- in org.apache.hadoop.fs.adl.live.TestMetadata
[INFO] Running org.apache.hadoop.fs.adl.TestAzureADTokenProvider
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.952 s
- in org.apache.hadoop.fs.adl.TestAzureADTokenProvider
[INFO] Running org.apache.hadoop.fs.adl.TestRelativePathFormation
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.1 s -
in org.apache.hadoop.fs.adl.TestRelativePathFormation
[INFO] Running org.apache.hadoop.fs.adl.TestValidateConfiguration
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.154 s
- in org.apache.hadoop.fs.adl.TestValidateConfiguration
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 869, Failures: 0, Errors: 0, Skipped: 4
{code}
> Upgrade to ADLS SDK which has major performance improvement for ingress/egress
> ------------------------------------------------------------------------------
>
> Key: HADOOP-15965
> URL: https://issues.apache.org/jira/browse/HADOOP-15965
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/adl
> Reporter: Vishwajeet Dusane
> Priority: Major
>
> Upgrade ADLS SDK to version 2.3.2 which has major improvements
> # Add special handling for 404 errors when requesting tokens from MSI
> # Fix liststatus response parsing when filestatus object contains array in
> one field.
> # Use wildfly openssl native binding with Java. This is a workaround to
> [https://bugs.openjdk.java.net/browse/JDK-8046943]issue. 2X performance boost
> over HTTPS. Similar to HADOOP-15669
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
