[ https://issues.apache.org/jira/browse/HUDI-269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16937358#comment-16937358 ]
Xing Pan edited comment on HUDI-269 at 9/25/19 2:56 PM: -------------------------------------------------------- [~vbalaji] yea, these strange 5K requests are mainly head requests, and cause a lot of s3 4xx error, which is defined as "client side error". I only have one partition "1100/01/01", and attached please find the *hudi_request_test.tar.gz* {code:java} aws s3 ls s3://xxx/output/1100/01/01/ 2019-09-25 01:56:57 93 .hoodie_partition_metadata 2019-09-25 02:12:18 535993 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-111-99_20190925021213.parquet 2019-09-25 02:50:30 679546 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-126-108_20190925025025.parquet 2019-09-25 02:32:27 597943 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-135-117_20190925023222.parquet 2019-09-25 02:38:03 623372 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-135-117_20190925023758.parquet 2019-09-25 02:12:48 537971 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-149-130_20190925021243.parquet 2019-09-25 02:50:39 680323 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-161-136_20190925025033.parquet 2019-09-25 02:32:57 599788 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-173-148_20190925023252.parquet 2019-09-25 02:38:33 625295 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-173-148_20190925023828.parquet 2019-09-25 02:13:18 540308 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-187-161_20190925021313.parquet 2019-09-25 02:50:47 681076 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-196-164_20190925025042.parquet 2019-09-25 02:31:07 591207 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-21-24_20190925023057.parquet 2019-09-25 02:36:48 615894 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-21-24_20190925023637.parquet 2019-09-25 02:50:01 675036 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-21-24_20190925024946.parquet 2019-09-25 02:33:27 602011 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-211-179_20190925023322.parquet 2019-09-25 02:39:03 627524 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-211-179_20190925023858.parquet 2019-09-25 02:13:48 542690 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-225-192_20190925021343.parquet 2019-09-25 02:50:55 681495 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-231-192_20190925025049.parquet 2019-09-25 02:33:57 604273 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-249-210_20190925023352.parquet 2019-09-25 02:39:33 629743 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-249-210_20190925023928.parquet 2019-09-25 02:14:18 545021 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-263-223_20190925021413.parquet 2019-09-25 02:51:03 682267 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-266-220_20190925025058.parquet 2019-09-25 02:34:27 606495 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-287-241_20190925023422.parquet 2019-09-25 02:40:03 632018 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-287-241_20190925023958.parquet 2019-09-25 02:51:11 682667 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-301-248_20190925025106.parquet 2019-09-25 02:14:48 547294 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-301-254_20190925021443.parquet 2019-09-25 02:34:57 608770 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-325-272_20190925023452.parquet 2019-09-25 02:40:33 634280 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-325-272_20190925024028.parquet 2019-09-25 02:51:18 683418 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-336-276_20190925025113.parquet 2019-09-25 02:15:18 549588 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-339-285_20190925021513.parquet 2019-09-25 01:56:59 533148 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-35-37_20190925015651.parquet 2019-09-25 02:35:27 610998 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-363-303_20190925023522.parquet 2019-09-25 02:41:04 636524 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-363-303_20190925024058.parquet 2019-09-25 02:51:26 683833 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-371-304_20190925025121.parquet 2019-09-25 02:15:48 551902 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-377-316_20190925021543.parquet 2019-09-25 02:35:57 613259 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-401-334_20190925023552.parquet 2019-09-25 02:41:33 638757 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-401-334_20190925024128.parquet 2019-09-25 02:51:34 684572 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-406-332_20190925025130.parquet 2019-09-25 02:16:18 553820 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-415-347_20190925021613.parquet 2019-09-25 02:42:03 641007 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-439-365_20190925024158.parquet 2019-09-25 02:51:42 684965 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-441-360_20190925025137.parquet 2019-09-25 02:16:48 556070 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-453-378_20190925021643.parquet 2019-09-25 02:51:49 685729 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-476-388_20190925025144.parquet 2019-09-25 02:42:33 643281 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-477-396_20190925024228.parquet 2019-09-25 02:17:18 558313 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-491-409_20190925021713.parquet 2019-09-25 02:51:57 686138 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-511-416_20190925025152.parquet 2019-09-25 02:43:03 645147 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-515-427_20190925024258.parquet 2019-09-25 02:17:48 560572 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-529-440_20190925021743.parquet 2019-09-25 02:52:04 686907 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-546-444_20190925025159.parquet 2019-09-25 02:43:33 647422 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-553-458_20190925024328.parquet 2019-09-25 02:50:13 678037 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-56-52_20190925025005.parquet 2019-09-25 02:18:18 562863 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-567-471_20190925021813.parquet 2019-09-25 02:31:27 593489 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-59-55_20190925023122.parquet 2019-09-25 02:37:04 618808 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-59-55_20190925023658.parquet 2019-09-25 02:44:03 649706 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-591-489_20190925024358.parquet 2019-09-25 02:18:48 565105 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-605-502_20190925021843.parquet 2019-09-25 02:44:33 651954 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-629-520_20190925024428.parquet 2019-09-25 02:19:18 567380 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-643-533_20190925021913.parquet 2019-09-25 02:45:03 654233 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-667-551_20190925024458.parquet 2019-09-25 02:19:48 569611 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-681-564_20190925021943.parquet 2019-09-25 02:45:33 656494 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-705-582_20190925024528.parquet 2019-09-25 02:20:18 570001 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-719-595_20190925022013.parquet 2019-09-25 02:11:48 533634 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-73-68_20190925021143.parquet 2019-09-25 02:46:03 658751 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-743-613_20190925024558.parquet 2019-09-25 02:24:48 571980 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-757-626_20190925022443.parquet 2019-09-25 02:46:33 661058 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-781-644_20190925024628.parquet 2019-09-25 02:25:18 573843 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-795-657_20190925022513.parquet 2019-09-25 02:47:03 663307 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-819-675_20190925024658.parquet 2019-09-25 02:25:48 576084 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-833-688_20190925022543.parquet 2019-09-25 02:47:33 665619 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-857-706_20190925024728.parquet 2019-09-25 02:26:18 578286 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-871-719_20190925022613.parquet 2019-09-25 02:48:03 667486 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-895-737_20190925024758.parquet 2019-09-25 02:26:48 578665 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-909-750_20190925022643.parquet 2019-09-25 02:50:21 679170 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-91-80_20190925025016.parquet 2019-09-25 02:48:33 669758 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-933-768_20190925024828.parquet 2019-09-25 02:28:18 579813 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-947-781_20190925022813.parquet 2019-09-25 02:31:58 595760 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-97-86_20190925023152.parquet 2019-09-25 02:37:34 621072 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-97-86_20190925023728.parquet 2019-09-25 02:49:03 672033 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-971-799_20190925024858.parquet 2019-09-25 02:28:48 582017 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-985-812_20190925022843.parquet{code} was (Author: xingxpan): [~vbalaji] yea, these strange 5K requests are mainly head requests, and cause a lot of s3 4xx error, which is defined as "client side error". I only have one partition "1100/01/01", and attached pleas find the *hudi_request_test.tar.gz* {code:java} aws s3 ls s3://xxx/output/1100/01/01/ 2019-09-25 01:56:57 93 .hoodie_partition_metadata 2019-09-25 02:12:18 535993 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-111-99_20190925021213.parquet 2019-09-25 02:50:30 679546 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-126-108_20190925025025.parquet 2019-09-25 02:32:27 597943 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-135-117_20190925023222.parquet 2019-09-25 02:38:03 623372 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-135-117_20190925023758.parquet 2019-09-25 02:12:48 537971 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-149-130_20190925021243.parquet 2019-09-25 02:50:39 680323 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-161-136_20190925025033.parquet 2019-09-25 02:32:57 599788 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-173-148_20190925023252.parquet 2019-09-25 02:38:33 625295 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-173-148_20190925023828.parquet 2019-09-25 02:13:18 540308 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-187-161_20190925021313.parquet 2019-09-25 02:50:47 681076 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-196-164_20190925025042.parquet 2019-09-25 02:31:07 591207 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-21-24_20190925023057.parquet 2019-09-25 02:36:48 615894 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-21-24_20190925023637.parquet 2019-09-25 02:50:01 675036 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-21-24_20190925024946.parquet 2019-09-25 02:33:27 602011 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-211-179_20190925023322.parquet 2019-09-25 02:39:03 627524 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-211-179_20190925023858.parquet 2019-09-25 02:13:48 542690 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-225-192_20190925021343.parquet 2019-09-25 02:50:55 681495 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-231-192_20190925025049.parquet 2019-09-25 02:33:57 604273 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-249-210_20190925023352.parquet 2019-09-25 02:39:33 629743 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-249-210_20190925023928.parquet 2019-09-25 02:14:18 545021 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-263-223_20190925021413.parquet 2019-09-25 02:51:03 682267 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-266-220_20190925025058.parquet 2019-09-25 02:34:27 606495 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-287-241_20190925023422.parquet 2019-09-25 02:40:03 632018 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-287-241_20190925023958.parquet 2019-09-25 02:51:11 682667 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-301-248_20190925025106.parquet 2019-09-25 02:14:48 547294 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-301-254_20190925021443.parquet 2019-09-25 02:34:57 608770 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-325-272_20190925023452.parquet 2019-09-25 02:40:33 634280 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-325-272_20190925024028.parquet 2019-09-25 02:51:18 683418 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-336-276_20190925025113.parquet 2019-09-25 02:15:18 549588 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-339-285_20190925021513.parquet 2019-09-25 01:56:59 533148 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-35-37_20190925015651.parquet 2019-09-25 02:35:27 610998 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-363-303_20190925023522.parquet 2019-09-25 02:41:04 636524 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-363-303_20190925024058.parquet 2019-09-25 02:51:26 683833 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-371-304_20190925025121.parquet 2019-09-25 02:15:48 551902 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-377-316_20190925021543.parquet 2019-09-25 02:35:57 613259 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-401-334_20190925023552.parquet 2019-09-25 02:41:33 638757 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-401-334_20190925024128.parquet 2019-09-25 02:51:34 684572 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-406-332_20190925025130.parquet 2019-09-25 02:16:18 553820 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-415-347_20190925021613.parquet 2019-09-25 02:42:03 641007 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-439-365_20190925024158.parquet 2019-09-25 02:51:42 684965 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-441-360_20190925025137.parquet 2019-09-25 02:16:48 556070 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-453-378_20190925021643.parquet 2019-09-25 02:51:49 685729 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-476-388_20190925025144.parquet 2019-09-25 02:42:33 643281 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-477-396_20190925024228.parquet 2019-09-25 02:17:18 558313 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-491-409_20190925021713.parquet 2019-09-25 02:51:57 686138 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-511-416_20190925025152.parquet 2019-09-25 02:43:03 645147 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-515-427_20190925024258.parquet 2019-09-25 02:17:48 560572 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-529-440_20190925021743.parquet 2019-09-25 02:52:04 686907 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-546-444_20190925025159.parquet 2019-09-25 02:43:33 647422 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-553-458_20190925024328.parquet 2019-09-25 02:50:13 678037 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-56-52_20190925025005.parquet 2019-09-25 02:18:18 562863 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-567-471_20190925021813.parquet 2019-09-25 02:31:27 593489 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-59-55_20190925023122.parquet 2019-09-25 02:37:04 618808 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-59-55_20190925023658.parquet 2019-09-25 02:44:03 649706 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-591-489_20190925024358.parquet 2019-09-25 02:18:48 565105 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-605-502_20190925021843.parquet 2019-09-25 02:44:33 651954 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-629-520_20190925024428.parquet 2019-09-25 02:19:18 567380 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-643-533_20190925021913.parquet 2019-09-25 02:45:03 654233 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-667-551_20190925024458.parquet 2019-09-25 02:19:48 569611 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-681-564_20190925021943.parquet 2019-09-25 02:45:33 656494 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-705-582_20190925024528.parquet 2019-09-25 02:20:18 570001 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-719-595_20190925022013.parquet 2019-09-25 02:11:48 533634 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-73-68_20190925021143.parquet 2019-09-25 02:46:03 658751 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-743-613_20190925024558.parquet 2019-09-25 02:24:48 571980 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-757-626_20190925022443.parquet 2019-09-25 02:46:33 661058 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-781-644_20190925024628.parquet 2019-09-25 02:25:18 573843 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-795-657_20190925022513.parquet 2019-09-25 02:47:03 663307 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-819-675_20190925024658.parquet 2019-09-25 02:25:48 576084 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-833-688_20190925022543.parquet 2019-09-25 02:47:33 665619 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-857-706_20190925024728.parquet 2019-09-25 02:26:18 578286 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-871-719_20190925022613.parquet 2019-09-25 02:48:03 667486 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-895-737_20190925024758.parquet 2019-09-25 02:26:48 578665 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-909-750_20190925022643.parquet 2019-09-25 02:50:21 679170 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-91-80_20190925025016.parquet 2019-09-25 02:48:33 669758 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-933-768_20190925024828.parquet 2019-09-25 02:28:18 579813 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-947-781_20190925022813.parquet 2019-09-25 02:31:58 595760 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-97-86_20190925023152.parquet 2019-09-25 02:37:34 621072 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-97-86_20190925023728.parquet 2019-09-25 02:49:03 672033 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-971-799_20190925024858.parquet 2019-09-25 02:28:48 582017 68d656cc-65a5-47f7-bf28-961315e718bc-0_0-985-812_20190925022843.parquet{code} > Provide ability to throttle DeltaStreamer sync runs > --------------------------------------------------- > > Key: HUDI-269 > URL: https://issues.apache.org/jira/browse/HUDI-269 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: deltastreamer > Reporter: BALAJI VARADARAJAN > Assignee: Xing Pan > Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: hudi_request_test.tar.gz, > image-2019-09-25-08-51-19-686.png > > Time Spent: 10m > Remaining Estimate: 0h > > Copied from [https://github.com/apache/incubator-hudi/issues/922] > In some scenario in our cluster, we may want delta streamer to slow down a > bit. > so it's nice to have a parameter to control the min sync interval of each > sync in continuous mode. > this param is default to 0, so this should not affect current logic. > minor pr: [#921|https://github.com/apache/incubator-hudi/pull/921] > the main reason we want to slow it down is that aws s3 is charged by s3 > get/put/list requests. we don't want to pay for too many requests for a > really slow change table. -- This message was sent by Atlassian Jira (v8.3.4#803005)