steveloughran commented on PR #6306:
URL: https://github.com/apache/hadoop/pull/6306#issuecomment-1831872123
testing locally. the shaded pom (still) complains about logging and doesn't
let us turn on httpclient logging through log4j files. Otherwise all good.
cloudstore programmatically turns on shaded httpclient logs if you set
-debug on the cli. How about we add a virtual log org.apache.hadoop.fs.s3a.http
which, if set to debug will tell s3afs to switch logging to debug behind the
scenes?
(note; test executed from uk against uswest2 over VPN; latency is due to the
size of the planet)
```
bin/hadoop fs -ls -R s3a://stevel--usw2-az1--x-s3/
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
drwxrwxrwx - stevel stevel 0 2023-11-29 13:07
s3a://stevel--usw2-az1--x-s3/user
drwxrwxrwx - stevel stevel 0 2023-11-29 13:07
s3a://stevel--usw2-az1--x-s3/user/stevel
drwxrwxrwx - stevel stevel 0 2023-11-29 13:07
s3a://stevel--usw2-az1--x-s3/user/stevel/target
drwxrwxrwx - stevel stevel 0 2023-11-29 13:07
s3a://stevel--usw2-az1--x-s3/user/stevel/target/test
drwxrwxrwx - stevel stevel 0 2023-11-29 13:07
s3a://stevel--usw2-az1--x-s3/user/stevel/target/test/data
drwxrwxrwx - stevel stevel 0 2023-11-29 13:07
s3a://stevel--usw2-az1--x-s3/user/stevel/target/test/data/Z5ozmhmq7a
drwxrwxrwx - stevel stevel 0 2023-11-29 13:07
s3a://stevel--usw2-az1--x-s3/user/stevel/target/test/data/9r43oqPJPq
drwxrwxrwx - stevel stevel 0 2023-11-29 13:07
s3a://stevel--usw2-az1--x-s3/user/stevel/target/test/data/Q3pU1HXsFY
drwxrwxrwx - stevel stevel 0 2023-11-29 13:07
s3a://stevel--usw2-az1--x-s3/test
2023-11-29 13:07:47,117 [shutdown-hook-0] INFO
statistics.IOStatisticsLogging
(IOStatisticsLogging.java:logIOStatisticsAtLevel(269)) - IOStatistics:
counters=((action_http_head_request=4)
(audit_request_execution=19)
(audit_span_creation=13)
(object_list_request=14)
(object_metadata_request=4)
(op_get_file_status=1)
(op_glob_status=1)
(op_list_status=10)
(store_io_request=19));
gauges=();
minimums=((action_http_head_request.min=152)
(object_list_request.min=147)
(op_get_file_status.min=0)
(op_glob_status.min=4)
(op_list_status.min=152));
maximums=((action_http_head_request.max=184)
(object_list_request.max=1038)
(op_get_file_status.max=0)
(op_glob_status.max=4)
(op_list_status.max=1048));
means=((action_http_head_request.mean=(samples=4, sum=679, mean=169.7500))
(object_list_request.mean=(samples=14, sum=3080, mean=220.0000))
(op_get_file_status.mean=(samples=1, sum=0, mean=0.0000))
(op_glob_status.mean=(samples=1, sum=4, mean=4.0000))
(op_list_status.mean=(samples=10, sum=3787, mean=378.7000)));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
