xccui commented on issue #8305: URL: https://github.com/apache/hudi/issues/8305#issuecomment-1494861495
Hi @danny0405, I looked into this again. You are right, `returnContent()` will release the connection. Actually, I was misled by the code. There will be two `PoolingHttpClientConnectionManager`s at runtime. ``` leaseConnection:306, PoolingHttpClientConnectionManager (com.amazonaws.thirdparty.apache.http.impl.conn) get:282, PoolingHttpClientConnectionManager$1 (com.amazonaws.thirdparty.apache.http.impl.conn) invoke:-1, GeneratedMethodAccessor24 (jdk.internal.reflect) invoke:43, DelegatingMethodAccessorImpl (jdk.internal.reflect) invoke:566, Method (java.lang.reflect) invoke:70, ClientConnectionRequestFactory$Handler (com.amazonaws.http.conn) get:-1, $Proxy51 (com.amazonaws.http.conn) execute:190, MainClientExec (com.amazonaws.thirdparty.apache.http.impl.execchain) execute:186, ProtocolExec (com.amazonaws.thirdparty.apache.http.impl.execchain) doExecute:185, InternalHttpClient (com.amazonaws.thirdparty.apache.http.impl.client) execute:83, CloseableHttpClient (com.amazonaws.thirdparty.apache.http.impl.client) execute:56, CloseableHttpClient (com.amazonaws.thirdparty.apache.http.impl.client) execute:72, SdkHttpClient (com.amazonaws.http.apache.client.impl) executeOneRequest:1346, AmazonHttpClient$RequestExecutor (com.amazonaws.http) executeHelper:1157, AmazonHttpClient$RequestExecutor (com.amazonaws.http) doExecute:814, AmazonHttpClient$RequestExecutor (com.amazonaws.http) executeWithTimer:781, AmazonHttpClient$RequestExecutor (com.amazonaws.http) execute:755, AmazonHttpClient$RequestExecutor (com.amazonaws.http) access$500:715, AmazonHttpClient$RequestExecutor (com.amazonaws.http) execute:697, AmazonHttpClient$RequestExecutionBuilderImpl (com.amazonaws.http) execute:561, AmazonHttpClient (com.amazonaws.http) execute:541, AmazonHttpClient (com.amazonaws.http) invoke:5456, AmazonS3Client (com.amazonaws.services.s3) invoke:5403, AmazonS3Client (com.amazonaws.services.s3) getObjectMetadata:1372, AmazonS3Client (com.amazonaws.services.s3) lambda$getObjectMetadata$10:2545, S3AFileSystem (org.apache.hadoop.fs.s3a) apply:-1, 497983073 (org.apache.hadoop.fs.s3a.S3AFileSystem$$Lambda$1189) retryUntranslated:414, Invoker (org.apache.hadoop.fs.s3a) retryUntranslated:377, Invoker (org.apache.hadoop.fs.s3a) getObjectMetadata:2533, S3AFileSystem (org.apache.hadoop.fs.s3a) getObjectMetadata:2513, S3AFileSystem (org.apache.hadoop.fs.s3a) s3GetFileStatus:3776, S3AFileSystem (org.apache.hadoop.fs.s3a) innerGetFileStatus:3688, S3AFileSystem (org.apache.hadoop.fs.s3a) lambda$getFileStatus$24:3556, S3AFileSystem (org.apache.hadoop.fs.s3a) apply:-1, 718057245 (org.apache.hadoop.fs.s3a.S3AFileSystem$$Lambda$2610) lambda$trackDurationOfOperation$5:499, IOStatisticsBinding (org.apache.hadoop.fs.statistics.impl) apply:-1, 2039613101 (org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding$$Lambda$1168) trackDuration:444, IOStatisticsBinding (org.apache.hadoop.fs.statistics.impl) trackDurationAndSpan:2337, S3AFileSystem (org.apache.hadoop.fs.s3a) trackDurationAndSpan:2356, S3AFileSystem (org.apache.hadoop.fs.s3a) getFileStatus:3554, S3AFileSystem (org.apache.hadoop.fs.s3a) lambda$getFileStatus$17:410, HoodieWrapperFileSystem (org.apache.hudi.common.fs) get:-1, 589863653 (org.apache.hudi.common.fs.HoodieWrapperFileSystem$$Lambda$2609) executeFuncWithTimeMetrics:114, HoodieWrapperFileSystem (org.apache.hudi.common.fs) getFileStatus:404, HoodieWrapperFileSystem (org.apache.hudi.common.fs) checkTableValidity:51, TableNotFoundException (org.apache.hudi.exception) <init>:137, HoodieTableMetaClient (org.apache.hudi.common.table) newMetaClient:689, HoodieTableMetaClient (org.apache.hudi.common.table) access$000:81, HoodieTableMetaClient (org.apache.hudi.common.table) build:770, HoodieTableMetaClient$Builder (org.apache.hudi.common.table) createMetaClient:277, StreamerUtil (org.apache.hudi.util) <init>:118, WriteProfile (org.apache.hudi.sink.partitioner.profile) <init>:44, DeltaWriteProfile (org.apache.hudi.sink.partitioner.profile) getWriteProfile:75, WriteProfiles (org.apache.hudi.sink.partitioner.profile) lambda$singleton$0:64, WriteProfiles (org.apache.hudi.sink.partitioner.profile) apply:-1, 401283836 (org.apache.hudi.sink.partitioner.profile.WriteProfiles$$Lambda$3189) computeIfAbsent:1134, HashMap (java.util) singleton:63, WriteProfiles (org.apache.hudi.sink.partitioner.profile) create:56, BucketAssigners (org.apache.hudi.sink.partitioner) open:122, BucketAssignFunction (org.apache.hudi.sink.partitioner) openFunction:34, FunctionUtils (org.apache.flink.api.common.functions.util) open:100, AbstractUdfStreamOperator (org.apache.flink.streaming.api.operators) open:55, KeyedProcessOperator (org.apache.flink.streaming.api.operators) initializeStateAndOpenOperators:107, RegularOperatorChain (org.apache.flink.streaming.runtime.tasks) restoreGates:726, StreamTask (org.apache.flink.streaming.runtime.tasks) call:-1, 704715276 (org.apache.flink.streaming.runtime.tasks.StreamTask$$Lambda$3036) call:55, StreamTaskActionExecutor$1 (org.apache.flink.streaming.runtime.tasks) restoreInternal:702, StreamTask (org.apache.flink.streaming.runtime.tasks) restore:669, StreamTask (org.apache.flink.streaming.runtime.tasks) run:-1, 619798876 (org.apache.flink.runtime.taskmanager.Task$$Lambda$2938) runWithSystemExitMonitoring:935, Task (org.apache.flink.runtime.taskmanager) restoreAndInvoke:904, Task (org.apache.flink.runtime.taskmanager) doRun:728, Task (org.apache.flink.runtime.taskmanager) run:550, Task (org.apache.flink.runtime.taskmanager) run:829, Thread (java.lang) ``` ``` leaseConnection:276, PoolingHttpClientConnectionManager (org.apache.hudi.org.apache.http.impl.conn) get:263, PoolingHttpClientConnectionManager$1 (org.apache.hudi.org.apache.http.impl.conn) execute:190, MainClientExec (org.apache.hudi.org.apache.http.impl.execchain) execute:184, ProtocolExec (org.apache.hudi.org.apache.http.impl.execchain) execute:88, RetryExec (org.apache.hudi.org.apache.http.impl.execchain) execute:110, RedirectExec (org.apache.hudi.org.apache.http.impl.execchain) doExecute:184, InternalHttpClient (org.apache.hudi.org.apache.http.impl.client) execute:82, CloseableHttpClient (org.apache.hudi.org.apache.http.impl.client) execute:107, CloseableHttpClient (org.apache.hudi.org.apache.http.impl.client) execute:55, CloseableHttpClient (org.apache.hudi.org.apache.http.impl.client) execute:151, Request (org.apache.hudi.org.apache.http.client.fluent) get:545, RemoteHoodieTableFileSystemView (org.apache.hudi.common.table.view) executeRequest:185, RemoteHoodieTableFileSystemView (org.apache.hudi.common.table.view) getPendingCompactionOperations:458, RemoteHoodieTableFileSystemView (org.apache.hudi.common.table.view) apply:-1, 2001188586 (org.apache.hudi.common.table.view.PriorityBasedFileSystemView$$Lambda$3253) execute:69, PriorityBasedFileSystemView (org.apache.hudi.common.table.view) getPendingCompactionOperations:231, PriorityBasedFileSystemView (org.apache.hudi.common.table.view) <init>:96, CleanPlanner (org.apache.hudi.table.action.clean) requestClean:105, CleanPlanActionExecutor (org.apache.hudi.table.action.clean) requestClean:150, CleanPlanActionExecutor (org.apache.hudi.table.action.clean) execute:176, CleanPlanActionExecutor (org.apache.hudi.table.action.clean) scheduleCleaning:310, HoodieFlinkCopyOnWriteTable (org.apache.hudi.table) scheduleTableServiceInternal:430, BaseHoodieTableServiceClient (org.apache.hudi.client) clean:543, BaseHoodieTableServiceClient (org.apache.hudi.client) clean:758, BaseHoodieWriteClient (org.apache.hudi.client) clean:730, BaseHoodieWriteClient (org.apache.hudi.client) lambda$open$0:67, CleanFunction (org.apache.hudi.sink) run:-1, 1349800153 (org.apache.hudi.sink.CleanFunction$$Lambda$3117) lambda$wrapAction$0:130, NonThrownExecutor (org.apache.hudi.sink.utils) run:-1, 329014366 (org.apache.hudi.sink.utils.NonThrownExecutor$$Lambda$3118) runWorker:1128, ThreadPoolExecutor (java.util.concurrent) run:628, ThreadPoolExecutor$Worker (java.util.concurrent) run:829, Thread (java.lang) ``` The one that caused our exception is the AWS shaded one. It's irrelevant to the `EXECUTOR` used by Hudi. I'll close this for now and get deeper into the connection pool's behaviour by collecting some debug logs. Really appreciate your insights! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
