Marton Elek created HDDS-2506:
---------------------------------
Summary: Remove keyAllocationInfo and replication info from the
auditLog
Key: HDDS-2506
URL: https://issues.apache.org/jira/browse/HDDS-2506
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: Ozone Manager
Reporter: Marton Elek
During the review of HDDS-2470 I found that the full keyLocationInfo is added
to the audit log for s3 operations:
{code:java}
2019-11-15 12:34:18,538 | INFO | OMAudit | user=hadoop | ip=192.168.16.2 |
op=ALLOCATE_KEY {volume=s3b607288814a5da737a92fb067500396e, bucket=bucket1,
key=key1, dataSize=3813, replicationType=RATIS, replicationFactor=ONE,
keyLocationInfo=[]} | ret=SUCCESS | 2019-11-15 12:34:20,576 | INFO | OMAudit
| user=hadoop | ip=192.168.16.2 | op=ALLOCATE_KEY
{volume=s3b607288814a5da737a92fb067500396e, bucket=bucket1, key=key1,
dataSize=3813, replicationType=RATIS, replicationFactor=ONE,
keyLocationInfo=[]} | ret=SUCCESS | 2019-11-15 12:34:20,626 | INFO | OMAudit
| user=hadoop | ip=192.168.16.2 | op=ALLOCATE_BLOCK
{volume=s3b607288814a5da737a92fb067500396e, bucket=bucket1, key=key1,
dataSize=3813, replicationType=RATIS, replicationFactor=THREE,
keyLocationInfo=[], clientID=103141950132977668} | ret=SUCCESS | 2019-11-15
12:34:51,705 | INFO | OMAudit | user=hadoop | ip=192.168.16.2 |
op=COMMIT_MULTIPART_UPLOAD_PARTKEY {volume=s3b607288814a5da737a92fb067500396e,
bucket=bucket1, key=key1, dataSize=3813, replicationType=RATIS,
replicationFactor=ONE, keyLocationInfo=[blockID { containerBlockID {
containerID: 1 localID: 103141950135009280 } blockCommitSequenceId:
2}offset: 0length: 3813createVersion: 0pipeline { members { uuid:
"eefe54e8-5723-458e-9204-207c6b97c9b3" ipAddress: "192.168.16.3"
hostName: "ozones3_datanode_1.ozones3_default" ports { name: "RATIS"
value: 9858 } ports { name: "STANDALONE" value: 9859 }
networkName: "eefe54e8-5723-458e-9204-207c6b97c9b3" networkLocation:
"/default-rack" } members { uuid: "ebf127d7-90a9-4f06-8fe5-a0c9c9adb743"
ipAddress: "192.168.16.7" hostName: "ozones3_datanode_2.ozones3_default"
ports { name: "RATIS" value: 9858 } ports { name:
"STANDALONE" value: 9859 } networkName:
"ebf127d7-90a9-4f06-8fe5-a0c9c9adb743" networkLocation: "/default-rack" }
members { uuid: "9979c326-4982-4a4c-b34e-e70c1d825f5f" ipAddress:
"192.168.16.6" hostName: "ozones3_datanode_3.ozones3_default" ports {
name: "RATIS" value: 9858 } ports { name: "STANDALONE"
value: 9859 } networkName: "9979c326-4982-4a4c-b34e-e70c1d825f5f"
networkLocation: "/default-rack" } state: PIPELINE_OPEN type: RATIS factor:
THREE id { id: "69ba305b-fe89-4f5c-97cd-b894d5ee8f2b" } leaderID: ""}],
partNumber=1,
partName=/s3b607288814a5da737a92fb067500396e/bucket1/key1103141950132977668} |
ret=SUCCESS | 2019-11-15 12:42:10,883 | INFO | OMAudit | user=hadoop |
ip=192.168.16.2 | op=COMPLETE_MULTIPART_UPLOAD
{volume=s3b607288814a5da737a92fb067500396e, bucket=bucket1, key=key1,
dataSize=0, replicationType=RATIS, replicationFactor=ONE, keyLocationInfo=[],
multipartList=[partNumber: 1partName:
"/s3b607288814a5da737a92fb067500396e/bucket1/key1103141950132977668"]} |
ret=SUCCESS |
{code}
Including the full keyLocation info in the audit log may cause some problems:
* It makes the the audit log slower
* It makes harder to parse the audit log
I think it's better to separate the debug log (which can be provided easily
with ozone insight tool) from the audit log. Therefore I suggest to remove the
keyLocationInfo, replicationType, replicationFactor from the aduit log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
