[
https://issues.apache.org/jira/browse/HADOOP-17511?focusedWorklogId=547798&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-547798
]
ASF GitHub Bot logged work on HADOOP-17511:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 04/Feb/21 19:17
Start Date: 04/Feb/21 19:17
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2675:
URL: https://github.com/apache/hadoop/pull/2675#issuecomment-773544133
Latest release adds operation context info to the referer logs so can
generate a trace mapping S3 operations to the FS ops and S3A FS instance making
the call. No tie-in to opentelemetry tho'. Future work.
```
183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8
stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10
arn:aws:iam::152813717728:user/stevel-dev 0E574923013D27E3 REST.DELETE.OBJECT
fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/.distcp.tmp.attempt_local1771928054_0008_m_000001_0.1612458639648
"DELETE
/fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/.distcp.tmp.attempt_local1771928054_0008_m_000001_0.1612458639648
HTTP/1.1" 204 - - - 14 -
"http://hadoop.apache.org/audit/0ce4df25-02cc-43d0-98a2-6b13f12662dd/span/17.08.39#343/operation/op_rename";
"Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.11.901 Mac_OS_X/10.15.7
OpenJDK_64-Bit_Server_VM/25.252-b09 java/1.8.0_252 vendor/AdoptOpenJDK" -
EE5czDIkLDNEPLxZ8XDADyRrZQ5KvstPk1E4Hei1qceDWnG+nAg7CrutshO9eEhmRhBuviommqY=
SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader
stevel-london.s3.eu-west-2.amazonaws.com TLSv1.2
183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8
stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10
arn:aws:iam::152813717728:user/stevel-dev 1CF7A2A5E1AE304E REST.HEAD.OBJECT
fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/subDir2/subDir2/newfile1
"HEAD
/fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/subDir2/subDir2/newfile1
HTTP/1.1" 404 NoSuchKey 417 - 8 -
"http://hadoop.apache.org/audit/0ce4df25-02cc-43d0-98a2-6b13f12662dd/span/17.08.39#341/operation/op_exists";
"Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.11.901 Mac_OS_X/10.15.7
OpenJDK_64-Bit_Server_VM/25.252-b09 java/1.8.0_252 vendor/AdoptOpenJDK" -
1FGjqpSb90398PGXq7yUMEWrJnWb5Gk8tGHXCj5adoFJxVleU0ffo93JQBJ/6kEL8+glLrpS0Ik=
SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader
stevel-london.s3.eu-west-2.amazonaws.com TLSv1.2
183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8
stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10
arn:aws:iam::152813717728:user/stevel-dev 243F9FE15CB25A9E REST.HEAD.OBJECT
fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/.distcp.tmp.attempt_local1771928054_0008_m_000001_0.1612458639648
"HEAD
/fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/.distcp.tmp.attempt_local1771928054_0008_m_000001_0.1612458639648
HTTP/1.1" 200 - - 0 5 - "-" "Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.11.901
Mac_OS_X/10.15.7 OpenJDK_64-Bit_Server_VM/25.252-b09 java/1.8.0_252
vendor/AdoptOpenJDK" -
h4p6Og/ROqiWR7THNlvl7jK8mWSNbWbne2HfUvi+HO88IsW/seDQ1o8KvTJ6+FxgBKbH2JgCW24=
SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader
stevel-london.s3.eu-west-2.amazonaws.com TLSv1.2
183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8
stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10
arn:aws:iam::152813717728:user/stevel-dev 470666E02E16F8B6 REST.GET.BUCKET -
"GET
/?list-type=2&delimiter=%2F&max-keys=2&prefix=fork-0003%2Ftest%2FITestS3AContractDistCp%2FtestUpdateDeepDirectoryStructureToRemote%2Fremote%2FDELAY_LISTING_ME%2FoutputDir%2FinputDir%2FsubDir2%2FsubDir2%2Fnewfile1%2F&fetch-owner=false
HTTP/1.1" 200 - 412 - 13 12
"http://hadoop.apache.org/audit/0ce4df25-02cc-43d0-98a2-6b13f12662dd/span/17.08.39#343/operation/op_rename";
"Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.11.901 Mac_OS_X/10.15.7
OpenJDK_64-Bit_Server_VM/25.252-b09 java/1.8.0_252 vendor/AdoptOpenJDK" -
p6rYqjQGRRT7jzz1tpdyshDJ673ul1A4YYHb/CFLOTv8++McNJYMxev+98vjEU1Kute5ggU3sFM=
SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader
stevel-london.s3.eu-west-2.amazonaws.com TLSv1.2
183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8
stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10
arn:aws:iam::152813717728:user/stevel-dev 5F2E21B0F0071218 REST.COPY.OBJECT
fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/subDir2/subDir2/newfile1
"PUT
/fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/subDir2/subDir2/newfile1
HTTP/1.1" 200 - 234 0 41 19
"http://hadoop.apache.org/audit/0ce4df25-02cc-43d0-98a2-6b13f12662dd/span/17.08.39#343/operation/op_rename";
"Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.11.901 Mac_OS_X/10.15.7
OpenJDK_64-Bit_Server_VM/25.252-b09 java/1.8.0_252 vendor/AdoptOpenJDK
com.amazonaws.services.s3.transfer.TransferManager/1.11.901" -
bkgJI8wkqzT6Cx+jsbLFB6Mra/qLJiwa0VxvginE5nk3xBT15dBp0Tn8m3HLom2oChaybsbUSFc=
SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader
stevel-london.s3.eu-west-2.amazonaws.com TLSv1.2
183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8
stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10
arn:aws:iam::152813717728:user/stevel-dev 5F2E21B0F0071218 REST.COPY.OBJECT_GET
fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/.distcp.tmp.attempt_local1771928054_0008_m_000001_0.1612458639648
- 200 - - 0 - - - - -
bkgJI8wkqzT6Cx+jsbLFB6Mra/qLJiwa0VxvginE5nk3xBT15dBp0Tn8m3HLom2oChaybsbUSFc=
SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader
stevel-london.s3.eu-west-2.amazonaws.com TLSv1.2
183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8
stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10
arn:aws:iam::152813717728:user/stevel-dev 82F908B5DE7BCF3D REST.HEAD.OBJECT
fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/.distcp.tmp.attempt_local1771928054_0008_m_000001_0.1612458639648
"HEAD
/fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/.distcp.tmp.attempt_local1771928054_0008_m_000001_0.1612458639648
HTTP/1.1" 200 - - 0 6 -
"http://hadoop.apache.org/audit/0ce4df25-02cc-43d0-98a2-6b13f12662dd/span/17.08.39#343/operation/op_rename";
"Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.11.901 Mac_OS_X/10.15.7
OpenJDK_64-Bit_Server_VM/25.252-b09 java/1.8.0_252 vendor/AdoptOpenJDK" -
4czEsj1JWzHooqkzqfCW2W/U0ofSs3KUpDUWD9HeYE2POs7AtUP9vG/wDWOCG/vMRZmogEOxMZs=
SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader
stevel-london.s3.eu-west-2.amazonaws.com TLSv1.2
183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8
stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10
arn:aws:iam::152813717728:user/stevel-dev 9774E0C9B42FB919 BATCH.DELETE.OBJECT
fork-0003/ - 204 - - - - - - - -
pbNOdYmZ24OIqstCyMnVcnT8PQlxJlYSnFSIiZrGLFjy6/UHFcySp78w0gaB0TgxRO2CJTakx+A=
SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader
stevel-london.s3.eu-west-2.amazonaws.com TLSv1.2
183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8
stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10
arn:aws:iam::152813717728:user/stevel-dev 9774E0C9B42FB919 BATCH.DELETE.OBJECT
fork-0003/test/ - 204 - - - - - - - -
pbNOdYmZ24OIqstCyMnVcnT8PQlxJlYSnFSIiZrGLFjy6/UHFcySp78w0gaB0TgxRO2CJTakx+A=
SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader
stevel-london.s3.eu-west-2.amazonaws.com TLSv1.2
183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8
stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10
arn:aws:iam::152813717728:user/stevel-dev 9774E0C9B42FB919 BATCH.DELETE.OBJECT
fork-0003/test/ITestS3AContractDistCp/ - 204 - - - - - - - -
pbNOdYmZ24OIqstCyMnVcnT8PQlxJlYSnFSIiZrGLFjy6/UHFcySp78w0gaB0TgxRO2CJTakx+A=
SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader stevel-london.s3.eu-west-
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 547798)
Time Spent: 3h 40m (was: 3.5h)
> Add an Audit plugin point for S3A auditing/context
> --------------------------------------------------
>
> Key: HADOOP-17511
> URL: https://issues.apache.org/jira/browse/HADOOP-17511
> Project: Hadoop Common
> Issue Type: Sub-task
> Affects Versions: 3.3.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Add a way for auditing tools to correlate S3 object calls with Hadoop FS API
> calls.
> Initially just to log/forward to an auditing service.
> Later: let us attach them as parameters in S3 requests, such as opentrace
> headeers or (my initial idea: http referrer header -where it will get into
> the log)
> Challenges
> * ensuring the audit span is created for every public entry point. That will
> have to include those used in s3guard tools, some defacto public APIs
> * and not re-entered for active spans. s3A code must not call back into the
> FS API points
> * Propagation across worker threads
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
