xichen01 commented on PR #744: URL: https://github.com/apache/ratis/pull/744#issuecomment-1248000331
I use this https://github.com/apache/ratis/pull/744/commits/f0a09fabf1eb488e5849a7dc105e6cc7047ca678 commit code to test Ozone S3, Tested with a multiupload of 100 threads, GRPC message metric leaks still found. This is the test code ```python3 import boto3 from boto3.s3.transfer import TransferConfig MB = 1024 * 1024 s3 = boto3.resource('s3') MultipartChunkSize = 5 * MB MaxConcurrency = 100 AccessKeyID = "chen" SecretAccessKey = "user_key" BucketRegion = "dummyregion" BucketName = "testbucket" EndpointUrl = "http://localhost:9878"; LocalFilePath = "/Users/pony.chen/500M.img" ObjectKeyName = "key_from_py1" def multipart_upload(client, local_file_path, bucket_name, object_key, multipart_chunksize, max_concurrency): config = TransferConfig(multipart_chunksize=multipart_chunksize, max_concurrency=max_concurrency) client.upload_file(local_file_path, bucket_name, object_key, Config=config) if __name__ == '__main__': client = boto3.client('s3', endpoint_url=EndpointUrl, aws_access_key_id=AccessKeyID, aws_secret_access_key=SecretAccessKey) multipart_upload(client, LocalFilePath, BucketName, ObjectKeyName, MultipartChunkSize, MaxConcurrency) ``` I found that the leaked metrics are all related `XXX_received_executed` metrics ```bash [chen@hostname /root/ratis]% curl -s http://localhost:9878/jmx | grep client_message_metrics "name" : "ratis:name=ratis.client_message_metrics.client-78FD5D54C056->0806d407-c2cb-4628-a986-9d9f4da7393f.client-78FD5D54C056->0806d407-c2cb-4628-a986-9d9f4da7393f_ratis.grpc.RaftClientProtocolService_ordered_OK_received_executed", "name" : "ratis:name=ratis.client_message_metrics.client-F3678E2734CB->3be0191f-50c8-4c86-9d31-b9d5169c42db.client-F3678E2734CB->3be0191f-50c8-4c86-9d31-b9d5169c42db_ratis.grpc.RaftClientProtocolService_unordered_OK_received_executed", "name" : "ratis:name=ratis.client_message_metrics.client-268EA6C4F3CA->0806d407-c2cb-4628-a986-9d9f4da7393f.client-268EA6C4F3CA->0806d407-c2cb-4628-a986-9d9f4da7393f_ratis.grpc.RaftClientProtocolService_ordered_OK_received_executed", "name" : "ratis:name=ratis.client_message_metrics.client-36DFCCCF5CE1->3be0191f-50c8-4c86-9d31-b9d5169c42db.client-36DFCCCF5CE1->3be0191f-50c8-4c86-9d31-b9d5169c42db_ratis.grpc.RaftClientProtocolService_ordered_OK_received_executed", [chen@hostname /root/ratis]% ``` So I added some `synchronized` at this commit https://github.com/apache/ratis/pull/744/commits/22d646a1b7cbed94def3c901e5a4e66874ea2be1 after applying this commit https://github.com/apache/ratis/pull/744/commits/22d646a1b7cbed94def3c901e5a4e66874ea2be1, all metrics leaks are gone But I don't really understand why this can fix the GRPC message metrics leak, any suggestions? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
