[
https://issues.apache.org/jira/browse/HADOOP-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13122:
-----------------------------------
Attachment: HADOOP-13122.001.patch
I'm attaching patch v001.
This sets up the AWS SDK {{ClientConfiguration}} to prepend the Hadoop version
number into the User-Agent. If a configuration property is specified, then it
also prepends the property value in front of that. This information is
additive, not a full replacement. The information included by the AWS SDK is
still present.
{{TestS3AConfiguration}} includes tests with and without the custom prefix. I
refactored a bit to share code with other tests that need to use reflection to
check AWS SDK internals.
In addition to the unit tests, I ran manual testing live against S3, with the
following in log4j.properties:
{code}
log4j.logger.org.apache.hadoop.fs.s3a=DEBUG
log4j.logger.org.apache.http=DEBUG
log4j.logger.org.apache.http.wire=ERROR
{code}
I could see the initialization log message announcing what it would use for the
User-Agent. Then, the HTTP Components log messages showed that the User-Agent
was in fact passing all the way through to the HTTP call:
h3. Before Patch
{code}
hadoop fs -ls s3a://cnauroth-test-aws-s3a/
User-Agent: aws-sdk-java/1.10.6 Mac_OS_X/10.9.5
Java_HotSpot(TM)_64-Bit_Server_VM/24.65-b04/1.7.0_67
{code}
h3. After Patch
{code}
hadoop fs -ls s3a://cnauroth-test-aws-s3a/
User-Agent: Hadoop 3.0.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.9.5
Java_HotSpot(TM)_64-Bit_Server_VM/24.65-b04/1.7.0_67
{code}
h3. After Patch/Custom Prefix:
{code}
hadoop fs -Dfs.s3a.user.agent.prefix=MyApp -ls s3a://cnauroth-test-aws-s3a/
User-Agent: MyApp, Hadoop 3.0.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.9.5
Java_HotSpot(TM)_64-Bit_Server_VM/24.65-b04/1.7.0_67
{code}
> Customize User-Agent header sent in HTTP requests by S3A.
> ---------------------------------------------------------
>
> Key: HADOOP-13122
> URL: https://issues.apache.org/jira/browse/HADOOP-13122
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: HADOOP-13122.001.patch
>
>
> S3A passes a User-Agent header to the S3 back-end. Right now, it uses the
> default value set by the AWS SDK, so Hadoop HTTP traffic doesn't appear any
> different from general AWS SDK traffic. If we customize the User-Agent
> header, then it will enable better troubleshooting and analysis by AWS or
> alternative providers of S3-like services.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
