[
https://issues.apache.org/jira/browse/FLINK-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee updated FLINK-13376:
---------------------------------
Description:
FileInputFormat use ContinuousFileReaderOperator in StreamExecutionEnvironment,
and ContinuousFileReaderOperator join the async thread in close, So before
closing, the data is incomplete.
But batch operators use endInput to do some work, this lead to incomplete
records in endInput, and lead to wrong results.
This mean batch mode of blink-planner can not use file related sources...
was:
FileInputFormat use ContinuousFileReaderOperator in StreamExecutionEnvironment,
and ContinuousFileReaderOperator join the async thread in close, So before
closing, the data is incomplete.
But batch operators use endInput to do some work, this lead to incomplete
records in endInput, and lead to wrong results.
> FileInputFormat can not be used in batch mode of blink-planner
> --------------------------------------------------------------
>
> Key: FLINK-13376
> URL: https://issues.apache.org/jira/browse/FLINK-13376
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / FileSystem, Runtime / Task, Table SQL /
> Runtime
> Reporter: Jingsong Lee
> Priority: Blocker
> Fix For: 1.9.0, 1.10.0
>
>
> FileInputFormat use ContinuousFileReaderOperator in
> StreamExecutionEnvironment, and ContinuousFileReaderOperator join the async
> thread in close, So before closing, the data is incomplete.
> But batch operators use endInput to do some work, this lead to incomplete
> records in endInput, and lead to wrong results.
> This mean batch mode of blink-planner can not use file related sources...
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
