morningman opened a new issue, #67446: URL: https://github.com/apache/doris/issues/67446
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master (`ddbaaab1388`), local single-FE + single-BE cluster, Iceberg REST catalog (`apache/iceberg-rest-fixture:1.10.0` + MinIO). ### What's Wrong? After a period of Iceberg activity, **every** subsequent Iceberg write on that FE fails with: ``` errCode = 2, detailMessage = java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@... [Not completed, task = ...Executors$RunnableAdapter@...[Wrapped task = org.apache.iceberg.util.Tasks$Builder$1@...]] rejected from java.util.concurrent.ThreadPoolExecutor@415866a0 [Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 26] ``` The executor is `Terminated`. `org.apache.iceberg.util.Tasks` submits to Iceberg's JVM-wide worker pool (`org.apache.iceberg.util.ThreadPools`), which is a static singleton — once it is shut down it stays down for the life of the JVM. Observed three times in one session. Every `insert into <iceberg table>` failed afterwards, including into freshly created catalogs, databases and tables. **Restarting the FE is the only recovery**; the pool identity and `completed tasks = 26` were identical across the failures. ### What You Expected? Iceberg writes keep working, or the pool is per-catalog and is not shared/shut down globally. ### How to Reproduce? I have **not** isolated the trigger. I explicitly ruled out three hypotheses — after each of these, an Iceberg write into a new catalog/database still succeeded: | Operation, then write again | Result | |---|---| | `drop catalog <iceberg catalog>` | works | | `drop database <db> force` | works | | a `CREATE TABLE` that fails (`Cannot add redundant partition`) | works | So the cause is still open. What the failing FE instance had in common: it had been up for ~1 hour and had run several Iceberg regression suites (`test_iceberg_write_partition_types_null`, `test_iceberg_write_partition_epoch_boundary`), including runs that failed midway. ### Anything Else? If the pool really is `ThreadPools.getWorkerPool()`, any code path that calls `shutdown()` on it (directly, or via an Iceberg API that closes a shared resource) would poison the whole FE. Grepping FE for `ThreadPools`, `shutdown()` on an Iceberg-owned executor, or `close()` on a shared `FileIO`/catalog object is probably the fastest way in. Filing this so the symptom is on record; happy to add more data if someone can suggest a narrowing experiment. Found while running Iceberg regression suites for #67366. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
