[
https://issues.apache.org/jira/browse/HADOOP-17628?focusedWorklogId=630485&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-630485
]
ASF GitHub Bot logged work on HADOOP-17628:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 28/Jul/21 12:35
Start Date: 28/Jul/21 12:35
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#issuecomment-888247513
Latest release
* Address review comments
* log IOStats after each test case.
Important: as the cached FS retains statistics, the numbers
get bigger over time.
* HDFS test is now reinstated, as we've identified that most
of its long execution time is from the large file upload/download
suites. Disable them and its execution time drops from 4m to 30s,
which means it can then be used to make sure the contract suite
is consistent between HDFS and the object stores.
IOStats of full suite against S3 london (1:43s)
```
2021-07-28 12:40:48,632 [setup] INFO statistics.IOStatisticsLogging
(IOStatisticsLogging.java:logIOStatisticsAtLevel(269)) - IOStatistics:
counters=((action_executor_acquired=47)
(action_http_get_request=38)
(action_http_head_request=111)
(audit_request_execution=420)
(audit_span_creation=483)
(directories_created=38)
(directories_deleted=1)
(fake_directories_deleted=485)
(files_copied=2)
(files_copied_bytes=264)
(files_created=47)
(files_deleted=48)
(ignored_errors=14)
(object_bulk_delete_request=88)
(object_copy_requests=2)
(object_delete_objects=534)
(object_delete_request=5)
(object_list_request=89)
(object_metadata_request=111)
(object_put_bytes=18880752)
(object_put_request=85)
(object_put_request_completed=85)
(op_create=47)
(op_delete=14)
(op_exists=13)
(op_exists.failures=3)
(op_get_file_status=194)
(op_get_file_status.failures=44)
(op_glob_status=25)
(op_is_file=1)
(op_list_files=9)
(op_list_status=60)
(op_mkdirs=64)
(op_open=39)
(op_rename=2)
(s3guard_metadatastore_initialization=1)
(s3guard_metadatastore_put_path_request=103)
(s3guard_metadatastore_record_deletes=2)
(s3guard_metadatastore_record_reads=1473)
(s3guard_metadatastore_record_writes=350)
(store_io_request=422)
(stream_read_bytes=18878052)
(stream_read_close_operations=39)
(stream_read_closed=38)
(stream_read_opened=38)
(stream_read_operations=2742)
(stream_read_operations_incomplete=1639)
(stream_read_seek_policy_changed=39)
(stream_read_total_bytes=18878052)
(stream_write_block_uploads=47)
(stream_write_bytes=18880752)
(stream_write_total_data=37761504));
gauges=((stream_write_block_uploads_pending=47));
minimums=((action_executor_acquired.min=0)
(action_http_get_request.min=31)
(action_http_head_request.min=22)
(object_bulk_delete_request.min=45)
(object_delete_request.min=34)
(object_list_request.min=28)
(object_put_request.min=42)
(op_create.min=16)
(op_delete.min=53)
(op_exists.failures.min=16)
(op_exists.min=15)
(op_get_file_status.failures.min=16)
(op_get_file_status.min=15)
(op_glob_status.min=15)
(op_is_file.min=43)
(op_list_files.min=176)
(op_list_status.min=64)
(op_mkdirs.min=16)
(op_rename.min=967));
maximums=((action_executor_acquired.max=0)
(action_http_get_request.max=123)
(action_http_head_request.max=317)
(object_bulk_delete_request.max=384)
(object_delete_request.max=91)
(object_list_request.max=202)
(object_put_request.max=2083)
(op_create.max=129)
(op_delete.max=2196)
(op_exists.failures.max=45)
(op_exists.max=43)
(op_get_file_status.failures.max=29)
(op_get_file_status.max=341)
(op_glob_status.max=192)
(op_is_file.max=43)
(op_list_files.max=589)
(op_list_status.max=260)
(op_mkdirs.max=729)
(op_rename.max=1199));
means=((action_executor_acquired.mean=(samples=47, sum=0, mean=0.0000))
(action_http_get_request.mean=(samples=38, sum=1490, mean=39.2105))
(action_http_head_request.mean=(samples=111, sum=4311, mean=38.8378))
(object_bulk_delete_request.mean=(samples=88, sum=12810, mean=145.5682))
(object_delete_request.mean=(samples=5, sum=260, mean=52.0000))
(object_list_request.mean=(samples=89, sum=4988, mean=56.0449))
(object_put_request.mean=(samples=85, sum=17463, mean=205.4471))
(op_create.mean=(samples=47, sum=1160, mean=24.6809))
(op_delete.mean=(samples=14, sum=11257, mean=804.0714))
(op_exists.failures.mean=(samples=3, sum=80, mean=26.6667))
(op_exists.mean=(samples=10, sum=250, mean=25.0000))
(op_get_file_status.failures.mean=(samples=44, sum=876, mean=19.9091))
(op_get_file_status.mean=(samples=150, sum=6404, mean=42.6933))
(op_glob_status.mean=(samples=25, sum=1826, mean=73.0400))
(op_is_file.mean=(samples=1, sum=43, mean=43.0000))
(op_list_files.mean=(samples=9, sum=3218, mean=357.5556))
(op_list_status.mean=(samples=60, sum=7084, mean=118.0667))
(op_mkdirs.mean=(samples=64, sum=15375, mean=240.2344))
(op_rename.mean=(samples=2, sum=2166, mean=1083.0000)));
```
IOStats of full suite against AWS cardiff (1:28). That region is about 30
miles away from here, though I don't know how cables are routed across the
Bristol Channel; it'll probably be a bit longer. In contrast, london will be
100-120 miles away, so latency always going to be a bit higher there.
```
2021-07-28 12:43:57,686 INFO [setup]: statistics.IOStatisticsLogging
(IOStatisticsLogging.java:logIOStatisticsAtLevel(269)) - IOStatistics:
counters=((action_http_delete_request=48)
(action_http_delete_request.failures=34)
(action_http_get_request=161)
(action_http_head_request=333)
(action_http_head_request.failures=79)
(action_http_put_request=237)
(bytes_received=18878316)
(bytes_sent=18881016)
(connections_made=779)
(directories_created=71)
(files_created=49)
(get_responses=779)
(op_create=49)
(op_delete=48)
(op_exists=53)
(op_get_file_status=291)
(op_list_status=107)
(op_mkdirs=71)
(op_open=41)
(op_rename=22)
(send_requests=237));
gauges=();
minimums=((action_http_delete_request.failures.min=21)
(action_http_delete_request.min=31)
(action_http_get_request.min=21)
(action_http_head_request.failures.min=20)
(action_http_head_request.min=19)
(action_http_put_request.min=23));
maximums=((action_http_delete_request.failures.max=332)
(action_http_delete_request.max=146)
(action_http_get_request.max=2193)
(action_http_head_request.failures.max=262)
(action_http_head_request.max=822)
(action_http_put_request.max=3370));
means=((action_http_delete_request.failures.mean=(samples=34, sum=1901,
mean=55.9118))
(action_http_delete_request.mean=(samples=14, sum=744, mean=53.1429))
(action_http_get_request.mean=(samples=161, sum=15025, mean=93.3230))
(action_http_head_request.failures.mean=(samples=79, sum=3668, mean=46.4304))
(action_http_head_request.mean=(samples=254, sum=9391, mean=36.9724))
(action_http_put_request.mean=(samples=237, sum=27099, mean=114.3418)));
```
ABFS is collecting many fewer stats, we really need
* duration of all FS API calls
* LIST performance numbers should be split from GET calls, which they
currently aren't.
Really interesting there that HEAD -> 404 has a mean time of 46ms; HEAD to
200 of 36 millis.
There's always going to be some probes before creating files, dirs, so that
negative cost
is going to be visible for those operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 630485)
Time Spent: 2h 50m (was: 2h 40m)
> Distcp contract test is really slow with ABFS and S3A; timing out
> -----------------------------------------------------------------
>
> Key: HADOOP-17628
> URL: https://issues.apache.org/jira/browse/HADOOP-17628
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, fs/s3, test, tools/distcp
> Affects Versions: 3.4.0
> Reporter: Bilahari T H
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> The test case testDistCpWithIterator in AbstractContractDistCpTest is
> consistently timing out.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
