GitHub user zuozhiw added a comment to the discussion: Parallelism bottleneck in multi-UDF workflows
Hi @jrt1899, thanks for raising this. I agree that waiting for so many records before flushing can significantly reduce pipelining benefits. I suggest the following order of improvements: 1. Add timer-based flushing. Build on Design 1, flush when either the batch fills or a configurable interval expires. Smaller batches on timeout are reasonable; Flink supports this through its [buffer timeout configuration](https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/config/#execution-buffer-timeout-interval). 2. Improve static partitioning first. If these UDFs process rows independently and require no key-based repartitioning, we can use one-to-one forwarding with equal parallelism—reducing 64 buffers to 8 in your example. With different parallelism, a fixed mapping like [Flink’s rescale](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/operators/overview/#rescaling) connects each upstream worker to a subset of downstream workers. So we don't need m x n full connections and have a much smaller number of one-to-one connections. 3. Evaluate dynamic routing afterward. Design 2 makes load-aware routing decisions at runtime. It could help with uneven workloads, but this is natrually more complex and would require a bit more design. E.g. [Flink’s FLIP-339](https://cwiki.apache.org/confluence/spaces/FLINK/pages/263425181/FLIP-339+Support+Adaptive+Partition+Selection+for+StreamPartitioner) is a useful reference. GitHub link: https://github.com/apache/texera/discussions/8519#discussioncomment-18523924 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
