[
https://issues.apache.org/jira/browse/FLINK-27377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17527996#comment-17527996
]
Yi Tang commented on FLINK-27377:
---------------------------------
Hi [~maguowei]
In my case, it blocks on rename operation as i mentioned on mail list [1].
You can try to insert a large number of rows (with almost several gigabytes
size) into a aliyun oss filesystem sink in batch mode (with SQL API of course).
You can try to reproduce it with the following statements:
{code:java}
SET 'execution.runtime-mode' = 'batch';
CREATE TABLE i (id BIGINT, name STRING) WITH ('connector'='datagen',
'number-of-rows'='20000000');
CREATE TABLE o (id BIGINT, name STRING) WITH ('connector'='filesystem',
'format'='csv', 'path'='oss://...');
INSERT INTO o SELECT * FROM i;
{code}
Thanks for your kind reminder about the contributing guide. It's a trial fix,
maybe i should mark it as draft.
Its main purpose is to help to explain this issue and why it blocks.
It's also a fix direction i think, the current public api provides the
FinalizeOnMaster hook, then flink should prevent from being blocked on any kind
of its customized implementation.
ref:
[1] [https://lists.apache.org/thread/0s686w1j9trtpx067zw6hdks4zmzstq5]
> Job master blocks on marking vertex finished
> --------------------------------------------
>
> Key: FLINK-27377
> URL: https://issues.apache.org/jira/browse/FLINK-27377
> Project: Flink
> Issue Type: Improvement
> Reporter: Yi Tang
> Priority: Minor
> Labels: pull-request-available
>
> I have a case using FileSystem as batch sink, it will commit files by
> {{FinalizeOnMaster#finalizeGlobal}} while updating state from Running to
> Finished.
> When {{FinalizeOnMaster#finalizeGlobal}} method is blocked, the
> {{JobMaster#updateTaskExecutionState}} will also be blocked. Then the job
> master can not respond any new requests until
> {{JobMaster#updateTaskExecutionState}} is complete.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
