Reo-LEI commented on issue #2918:
URL: https://github.com/apache/iceberg/issues/2918#issuecomment-926804817
> How does this scenario work for other sinks (like MySQL)? How can other
sinks ensure the processing ordering?
The `JdbcDynamicTableSink` and `KafkaDynamicSink` will return
`OutputFormatProvider` and `SinkFunctionProvider` as `SinkRuntimeProvider`.
These `SinkRuntimeProvider` will accept `OutputFormat` or `SinkFunction` and
append sink operator to upstream directly. So there is no `Filter` operator
between upstream and sink operator.
Therefore, the other sinks haven't the Disorder problem. But Iceberg
FlinkSink need accept upstream DataStream and append writer, commiter operators
to it instead of a single sink operator. So we cann't use the
`OutputFormatProvider` and `SinkFunctionProvider` API to avoid the `Filter`
operator added between upstream and FlinkSink.
But Fortunately, I think we could ignore the Disorder problem for FlinkSQL.
Because with the upgrade of flink to
1.13(https://github.com/apache/iceberg/issues/2558), The `CommonPhysicalSink`
has been remove and refactor to `CommonExecSink`. We can found the `Filter`
operator parallelism will be set to be the same as its upstream. So I think we
can no longer modify the `Filter` operator parallelism, and resolve the
Disorder problem by upgrade flink to 1.13.
PS:
https://github.com/apache/flink/blob/5f007ff6c8224c6e5a14de2c79296eb85a22fe1f/flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecSink.java#L211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
