infzo commented on issue #37136:
URL: https://github.com/apache/arrow/issues/37136#issuecomment-1675778264
> Would you mind set log level and see the logging of s3? I don't know how
time spends here, need more log to see why initialize is slow
I tried it, and it seems that the retry connection timed out:
169.254.169.254:80.
```
start = time.time()
pyarrow.fs.initialize_s3(fs.S3LogLevel.Debug)
obs_fs = fs.S3FileSystem(access_key="ak", secret_key="sk",
endpoint_override="ep")
pyarrow.fs.finalize_s3()
print(f'S3FileSystem cost {time.time() - start:.6f} s')
[DEBUG] 2023-08-12 07:59:36.598 task-scheduler [140450938611456]
id=0x7fbd40000b80: Scheduling epoll_event_loop_unsubscribe_cleanup task for
immediate execution
[DEBUG] 2023-08-12 07:59:36.598 task-scheduler [140444465874688]
id=0x7fbd40000b80: Running epoll_event_loop_unsubscribe_cleanup task with
<Canceled> status
[DEBUG] 2023-08-12 07:59:36.623 HttpClientFactory [140463560197952]
Initializing Http Static State
[DEBUG] 2023-08-12 07:59:36.623 HttpClientFactory [140463560197952]
Initializing Curl Http Client
[DEBUG] 2023-08-12 07:59:36.624 ClientConfiguration [140463560197952]
ClientConfiguration will use SDK Auto Resolved profile: [default] if not
specified by users.
[WARN] 2023-08-12 07:59:36.624 ClientConfiguration [140463560197952] Retry
Strategy will use the default max attempts.
[DEBUG] 2023-08-12 07:59:36.624 CurlHandleContainer [140463560197952]
Attempting to acquire curl connection.
[DEBUG] 2023-08-12 07:59:36.624 CurlHandleContainer [140463560197952] No
current connections available in pool. Attempting to create new connections.
[DEBUG] 2023-08-12 07:59:36.624 CurlHandleContainer [140463560197952]
attempting to grow pool size by 2
[INFO] 2023-08-12 07:59:36.624 CurlHandleContainer [140463560197952] Pool
grown by 2
[INFO] 2023-08-12 07:59:36.624 CurlHandleContainer [140463560197952]
Connection has been released. Continuing.
[DEBUG] 2023-08-12 07:59:36.624 CurlHandleContainer [140463560197952]
Returning connection handle 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:36.624 CurlHttpClient [140463560197952] Obtained
connection handle 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:36.624 CURL [140463560197952] (Text) Trying
169.254.169.254:80...
[DEBUG] 2023-08-12 07:59:37.625 CURL [140463560197952] (Text) ipv4 connect
timeout after 1000ms, move on!
[DEBUG] 2023-08-12 07:59:37.625 CURL [140463560197952] (Text) Failed to
connect to 169.254.169.254 port 80 after 1001 ms: Timeout was reached
[DEBUG] 2023-08-12 07:59:37.625 CURL [140463560197952] (Text) Closing
connection 0
[ERROR] 2023-08-12 07:59:37.625 CurlHttpClient [140463560197952] Curl
returned error code 28 - Timeout was reached
[DEBUG] 2023-08-12 07:59:37.625 CurlHandleContainer [140463560197952]
Destroy curl handle: 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:37.625 CurlHandleContainer [140463560197952]
Created replacement handle and released to pool: 0x55ffc94fe330
[ERROR] 2023-08-12 07:59:37.625 EC2MetadataClient [140463560197952] Http
request to retrieve credentials failed
[WARN] 2023-08-12 07:59:37.625 EC2MetadataClient [140463560197952] Request
failed, now waiting 0 ms before attempting again.
[DEBUG] 2023-08-12 07:59:37.625 CurlHandleContainer [140463560197952]
Attempting to acquire curl connection.
[INFO] 2023-08-12 07:59:37.625 CurlHandleContainer [140463560197952]
Connection has been released. Continuing.
[DEBUG] 2023-08-12 07:59:37.625 CurlHandleContainer [140463560197952]
Returning connection handle 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:37.625 CurlHttpClient [140463560197952] Obtained
connection handle 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:37.625 CURL [140463560197952] (Text) Trying
169.254.169.254:80...
[DEBUG] 2023-08-12 07:59:38.626 CURL [140463560197952] (Text) ipv4 connect
timeout after 1000ms, move on!
[DEBUG] 2023-08-12 07:59:38.626 CURL [140463560197952] (Text) Failed to
connect to 169.254.169.254 port 80 after 1001 ms: Timeout was reached
[DEBUG] 2023-08-12 07:59:38.626 CURL [140463560197952] (Text) Closing
connection 0
[ERROR] 2023-08-12 07:59:38.626 CurlHttpClient [140463560197952] Curl
returned error code 28 - Timeout was reached
[DEBUG] 2023-08-12 07:59:38.627 CurlHandleContainer [140463560197952]
Destroy curl handle: 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:38.627 CurlHandleContainer [140463560197952]
Created replacement handle and released to pool: 0x55ffc94fe330
[ERROR] 2023-08-12 07:59:38.627 EC2MetadataClient [140463560197952] Http
request to retrieve credentials failed
[ERROR] 2023-08-12 07:59:38.627 EC2MetadataClient [140463560197952] Can not
retrieve resource from http://169.254.169.254/latest/api/token
[DEBUG] 2023-08-12 07:59:38.627 CurlHandleContainer [140463560197952]
Attempting to acquire curl connection.
[INFO] 2023-08-12 07:59:38.627 CurlHandleContainer [140463560197952]
Connection has been released. Continuing.
[DEBUG] 2023-08-12 07:59:38.627 CurlHandleContainer [140463560197952]
Returning connection handle 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:38.627 CurlHttpClient [140463560197952] Obtained
connection handle 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:38.627 CURL [140463560197952] (Text) Trying
169.254.169.254:80...
[DEBUG] 2023-08-12 07:59:39.628 CURL [140463560197952] (Text) ipv4 connect
timeout after 1000ms, move on!
[DEBUG] 2023-08-12 07:59:39.628 CURL [140463560197952] (Text) Failed to
connect to 169.254.169.254 port 80 after 1001 ms: Timeout was reached
[DEBUG] 2023-08-12 07:59:39.628 CURL [140463560197952] (Text) Closing
connection 0
[ERROR] 2023-08-12 07:59:39.628 CurlHttpClient [140463560197952] Curl
returned error code 28 - Timeout was reached
[DEBUG] 2023-08-12 07:59:39.628 CurlHandleContainer [140463560197952]
Destroy curl handle: 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:39.628 CurlHandleContainer [140463560197952]
Created replacement handle and released to pool: 0x55ffc94fe330
[ERROR] 2023-08-12 07:59:39.628 EC2MetadataClient [140463560197952] Http
request to retrieve credentials failed
[WARN] 2023-08-12 07:59:39.628 EC2MetadataClient [140463560197952] Request
failed, now waiting 0 ms before attempting again.
[DEBUG] 2023-08-12 07:59:39.628 CurlHandleContainer [140463560197952]
Attempting to acquire curl connection.
[INFO] 2023-08-12 07:59:39.628 CurlHandleContainer [140463560197952]
Connection has been released. Continuing.
[DEBUG] 2023-08-12 07:59:39.628 CurlHandleContainer [140463560197952]
Returning connection handle 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:39.628 CurlHttpClient [140463560197952] Obtained
connection handle 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:39.628 CURL [140463560197952] (Text) Trying
169.254.169.254:80...
[DEBUG] 2023-08-12 07:59:40.629 CURL [140463560197952] (Text) ipv4 connect
timeout after 1000ms, move on!
[DEBUG] 2023-08-12 07:59:40.629 CURL [140463560197952] (Text) Failed to
connect to 169.254.169.254 port 80 after 1001 ms: Timeout was reached
[DEBUG] 2023-08-12 07:59:40.629 CURL [140463560197952] (Text) Closing
connection 0
[ERROR] 2023-08-12 07:59:40.629 CurlHttpClient [140463560197952] Curl
returned error code 28 - Timeout was reached
[DEBUG] 2023-08-12 07:59:40.629 CurlHandleContainer [140463560197952]
Destroy curl handle: 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:40.629 CurlHandleContainer [140463560197952]
Created replacement handle and released to pool: 0x55ffc94fe330
[ERROR] 2023-08-12 07:59:40.629 EC2MetadataClient [140463560197952] Http
request to retrieve credentials failed
[ERROR] 2023-08-12 07:59:40.629 EC2MetadataClient [140463560197952] Can not
retrieve resource from
http://169.254.169.254/latest/meta-data/iam/security-credentials
[DEBUG] 2023-08-12 07:59:40.629 CurlHandleContainer [140463560197952]
Attempting to acquire curl connection.
[INFO] 2023-08-12 07:59:40.629 CurlHandleContainer [140463560197952]
Connection has been released. Continuing.
[DEBUG] 2023-08-12 07:59:40.629 CurlHandleContainer [140463560197952]
Returning connection handle 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:40.629 CurlHttpClient [140463560197952] Obtained
connection handle 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:40.629 CURL [140463560197952] (Text) Trying
169.254.169.254:80...
[DEBUG] 2023-08-12 07:59:41.631 CURL [140463560197952] (Text) ipv4 connect
timeout after 1000ms, move on!
[DEBUG] 2023-08-12 07:59:41.631 CURL [140463560197952] (Text) Failed to
connect to 169.254.169.254 port 80 after 1001 ms: Timeout was reached
[DEBUG] 2023-08-12 07:59:41.631 CURL [140463560197952] (Text) Closing
connection 0
[ERROR] 2023-08-12 07:59:41.631 CurlHttpClient [140463560197952] Curl
returned error code 28 - Timeout was reached
[DEBUG] 2023-08-12 07:59:41.631 CurlHandleContainer [140463560197952]
Destroy curl handle: 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:41.631 CurlHandleContainer [140463560197952]
Created replacement handle and released to pool: 0x55ffc94fe330
[ERROR] 2023-08-12 07:59:41.631 EC2MetadataClient [140463560197952] Http
request to retrieve credentials failed
[WARN] 2023-08-12 07:59:41.631 EC2MetadataClient [140463560197952] Request
failed, now waiting 0 ms before attempting again.
[DEBUG] 2023-08-12 07:59:41.631 CurlHandleContainer [140463560197952]
Attempting to acquire curl connection.
[INFO] 2023-08-12 07:59:41.631 CurlHandleContainer [140463560197952]
Connection has been released. Continuing.
[DEBUG] 2023-08-12 07:59:41.631 CurlHandleContainer [140463560197952]
Returning connection handle 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:41.631 CurlHttpClient [140463560197952] Obtained
connection handle 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:41.631 CURL [140463560197952] (Text) Trying
169.254.169.254:80...
[DEBUG] 2023-08-12 07:59:42.632 CURL [140463560197952] (Text) ipv4 connect
timeout after 1000ms, move on!
[DEBUG] 2023-08-12 07:59:42.632 CURL [140463560197952] (Text) Failed to
connect to 169.254.169.254 port 80 after 1001 ms: Timeout was reached
[DEBUG] 2023-08-12 07:59:42.632 CURL [140463560197952] (Text) Closing
connection 0
[ERROR] 2023-08-12 07:59:42.632 CurlHttpClient [140463560197952] Curl
returned error code 28 - Timeout was reached
[DEBUG] 2023-08-12 07:59:42.632 CurlHandleContainer [140463560197952]
Destroy curl handle: 0x55ffc94fe330
[DEBUG] 2023-08-12 07:59:42.632 CurlHandleContainer [140463560197952]
Created replacement handle and released to pool: 0x55ffc94fe330
[ERROR] 2023-08-12 07:59:42.632 EC2MetadataClient [140463560197952] Http
request to retrieve credentials failed
[ERROR] 2023-08-12 07:59:42.632 EC2MetadataClient [140463560197952] Can not
retrieve resource from
http://169.254.169.254/latest/meta-data/placement/availability-zone
[INFO] 2023-08-12 07:59:42.632 EC2MetadataClient [140463560197952] Unable to
pull region from instance metadata service
[INFO] 2023-08-12 07:59:42.632 CurlHandleContainer [140463560197952]
Initializing CurlHandleContainer with size 25
S3FileSystem cost 6.046947 s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
