ChenSammi commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-888766424
One of our customers reports OM out of service because of very heavy GC.
After some investigation, they are uploading objects through s3g periodically
with same names, and in the mean while, fetching the objects corrently. Here
is top part of the jstack output.
num #instances #bytes class name
----------------------------------------------
1: 56349154 3213812672 [Ljava.lang.Object;
2: 54898991 2635151568
org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo
3: 54907944 1757054208 java.util.HashMap$Node
4: 54898991 1756767712
org.apache.hadoop.hdds.client.ContainerBlockID
5: 21320616 1535084352
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$KeyLocation
6: 56349257 1352382168 java.util.ArrayList
7: 54898991 1317575784 org.apache.hadoop.hdds.client.BlockID
8: 21320616 1193954496
org.apache.hadoop.hdds.protocol.proto.HddsProtos$ContainerBlockID
9: 21320617 1023389616
org.apache.hadoop.hdds.protocol.proto.HddsProtos$BlockID
The total heap size is 18G+. The root cause is just the same as your
cases.
After the issue is workaround by deleting some unsued objects, the customer
raised a requirement to disable the versioning on the bucket.
I checked the AWS S3 behavior, bucket versioning is by default disabled.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html.
I'm glad that we have started to solve this problem, for example, HDDS-5472,
HDDS-5393, HDDS-5461 and another PR to control the client side. I totally
agree that we can do it step by step.
For this patch, my +1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
