[
https://issues.apache.org/jira/browse/FLINK-31252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee updated FLINK-31252:
---------------------------------
Description:
The following batch assignment operation is for two things:
1. It can be evenly distributed during batch reading to avoid scheduling
problems (for example, the current resource can only schedule part of the
tasks) that cause some tasks to fail to read data.
2. Read with limit, if split is assigned one by one, it may cause the task to
repeatedly create SplitFetchers. After the task is created, it is found that it
is idle and then closed. Then, new split coming, it will create SplitFetcher
and repeatedly read the data of the limit number (the limit status is in the
SplitFetcher).
was:
{code:java}
// The following batch assignment operation is for two things:
// 1. It can be evenly distributed during batch reading to avoid
scheduling problems (for
// example, the current resource can only schedule part of the tasks)
that cause some tasks
// to fail to read data.
// 2. Read with limit, if split is assigned one by one, it may cause
the task to repeatedly
// create SplitFetchers. After the task is created, it is found that it
is idle and then
// closed. Then, new split coming, it will create SplitFetcher and
repeatedly read the data
// of the limit number (the limit status is in the SplitFetcher).
{code}
> Improve StaticFileStoreSplitEnumerator to assign batch splits
> -------------------------------------------------------------
>
> Key: FLINK-31252
> URL: https://issues.apache.org/jira/browse/FLINK-31252
> Project: Flink
> Issue Type: Improvement
> Components: Table Store
> Reporter: Jingsong Lee
> Assignee: Jingsong Lee
> Priority: Major
> Fix For: table-store-0.4.0
>
>
> The following batch assignment operation is for two things:
> 1. It can be evenly distributed during batch reading to avoid scheduling
> problems (for example, the current resource can only schedule part of the
> tasks) that cause some tasks to fail to read data.
> 2. Read with limit, if split is assigned one by one, it may cause the task to
> repeatedly create SplitFetchers. After the task is created, it is found that
> it is idle and then closed. Then, new split coming, it will create
> SplitFetcher and repeatedly read the data of the limit number (the limit
> status is in the SplitFetcher).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
