Below0 opened a new issue, #15827:
URL: https://github.com/apache/iceberg/issues/15827
### Apache Iceberg version
main (unreleased)
### Query engine
Flink
### Please describe the bug 🐞
In `DynamicIcebergSink.Builder.append()`, the `DynamicRecordProcessor`
(generator) operator does not have its parallelism explicitly set. It always
falls back to the environment's default parallelism, ignoring any parallelism
configured on the upstream operator.
```
Upstream source (parallelism=16)
↓
DynamicRecordProcessor (parallelism=env default) ← upstream ignored
↓
Sink Writer (parallelism=writeParallelism)
```
The existing `writeParallelism` option only applies to the sink writer
operator, not the generator.
**Current Workaround**
`env.setParallelism(N)` can be used, but it applies globally to all
operators (generator, updater, sink writer), making fine-grained control
impossible.
**Proposal**
Inherit upstream parallelism automatically, consistent with typical Flink
operator chaining behavior:
```java
converted.setParallelism(input.getParallelism());
```
When upstream has explicit parallelism, the generator follows it. When not
set (`-1`), it falls back to the env default — same as current behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
