sakuray opened a new issue, #2780: URL: https://github.com/apache/incubator-paimon/issues/2780
### Search before asking - [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar. ### Paimon version Paimon 0.6.0 ### Compute Engine Flink1.18.0 ### Minimal reproduce step 1.use flink sql-client.sh 2.not move paimon jar into flink/lib 3.execute follow sql many times create table Orders ( order_number bigint, price decimal(32,2), order_time timestamp(3) ) with ( 'connector' = 'datagen' ); add jar 'C:\flink_lib\hadoop-mapreduce-client-core-2.10.1.jar'; add jar 'C:\flink_lib\hadoop-hdfs-client-2.8.5.jar'; add jar 'C:\flink_lib\paimon-flink-1.18-0.6.0-incubating.jar'; add jar 'C:\flink_lib\paimon-s3-0.6.0-incubating.jar'; create table paimon_order ( order_number bigint, price decimal(32,2), order_time timestamp(3) ) with ( 'connector' = 'paimon', 'path' = 's3://local-test/paimon_order/', 's3.endpoint' = 'http://127.0.0.1:9000/', 's3.access-key' = '', 's3.secret-key' = '', 'bucket' = '1', 'auto-create'= 'true' ); insert into paimon_order select order_number, price, order_time from Orders; ### What doesn't meet your expectations? now, execute it once, metaspace use more memory, util oom. I find it maybe caused by S3FileIO, paimon just create S3AFileSystem, but not close it. thread dump we can see s3a-file thread and java-sdk-http-connection-reaper thread, this all create by new S3AFileSystem().initialize(). s3a-file-thread can set env.java.opts.taskmanager: -Dhadoop.metrics.init.mode=STANDBY to avoid init, but java-sdk-http-connection-reaper thread has no config can avoid init it. both of threads can't finish when task finish or crash or cancel, it caused childFirstClassLoader can't release memory. and i see S3AFileSystem provide close method to close both thread, but i can't get it to execute so any idea to fix it. by the way, I'm not sure if executing the close method will fix this issue, But now Mat analyzes the current impact of this ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
