suxiaogang223 opened a new pull request, #70793:
URL: https://github.com/apache/doris/pull/70793

   ### Background
   
   The Iceberg connector uses the SDK's shared worker pool for parallel 
manifest planning and manifest writes. All Iceberg catalogs on an FE, including 
Iceberg siblings of Hive catalogs, use the same plugin and pool. The SDK sizes 
this pool from a JVM property or environment variable, defaulting to max(2, 
available processors).
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: None
   PR Type: Enhancement
   
   Problem Summary:
   
   Operators need a deployment-level way to set the shared Iceberg worker 
budget through the connector configuration. Master already shares the SDK pool 
across catalogs, but iceberg.conf does not expose its size. Add a startup-only 
setting and apply it before the first connector can initialize the SDK pool, 
covering both scan planning and SDK-internal manifest writes.
   
   ### Details
   
   - **Before and after:** Previously only the SDK JVM property/environment 
setting controlled the shared pool. Operators can now configure its total 
worker count in iceberg.conf; no per-catalog executor is introduced.
   - **Configs and defaults:** Add optional `worker_thread_pool_size` to the 
Iceberg plugin's `iceberg.conf`. It must be a positive integer and requires an 
FE restart on each node. When unset or blank, preserve 
`iceberg.worker.num-threads` / `ICEBERG_WORKER_NUM_THREADS` and the SDK default 
of max(2, available processors). An explicit plugin setting takes precedence 
over those SDK settings. The SDK queue and separate engine metadata, deletion, 
authentication-refresh and cancellation-cleanup pools are unchanged.
   - **Impact on existing users:** Untouched deployments retain their existing 
worker count. Explicit settings apply across standalone Iceberg catalogs and 
Hive catalog Iceberg siblings. Invalid values fail connector initialization 
with the setting name. An already initialized SDK pool with a different size is 
rejected rather than silently ignoring the configured limit.
   - **Behavior change:** An explicit plugin setting changes the shared SDK 
worker count for manifest planning and manifest writes. SDK ownership, catalog 
authentication, worker classloader pinning and scan cancellation/cleanup are 
preserved.
   
   ### Testing
   
   Local checks passed:
   
   - `cd fe && mvn -B -pl :fe-connector-iceberg checkstyle:check 
-Dmaven.build.cache.enabled=false` — zero Checkstyle violations, including test 
sources.
   - `bash build-support/check-fe-connector-imports.sh`
   - `bash build-support/check-fe-core-metadata-funnel.sh`
   - `git diff --cached --check`
   
   Added JUnit coverage for positive/invalid/blank configuration, concurrent 
initialization across catalog contexts, SDK setting precedence, shared-pool 
survival after scan cancellation/close, worker classloader pinning and 
premature SDK initialization. These tests have not been executed locally. 
Compilation and unit/regression tests were not run; local validation was 
limited to style and static checks.
   
   ### Release note
   
   Iceberg's shared FE worker pool can be sized with the optional positive 
`worker_thread_pool_size` setting in the plugin's `iceberg.conf`. The setting 
applies to all Iceberg catalogs on that FE and requires an FE restart. Leaving 
it unset preserves the existing SDK configuration and defaults.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details).
   - Does this need documentation: No additional documentation; usage, 
defaults, precedence and restart requirements are documented in the shipped 
iceberg.conf.template.
   - Pre-submission status: Partially verified: local style/static checks 
passed; compilation and unit/regression tests have not been run.
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6).
   - Generated: Code, tests and this description.
   - Human review: Pending.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to