[
https://issues.apache.org/jira/browse/FLINK-29295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee updated FLINK-29295:
---------------------------------
Description:
In AbstractTableWrite, the Writer is cleaned up as soon as no new files are
generated, which may lead to the changes generated after the compaction have
not been committed, but the new data from the next checkpoint comes to create a
new writer, which conflicts with the changes generated in the next round of
checkpoint and the previous round, resulting in an exception:
{code:java}
Caused by: java.lang.IllegalStateException: Trying to delete file
{org.apache.flink.table.data.binary.BinaryRowData@5759f99e, 0, 0,
data-7bf2498e-d0a1-42a5-97b7-b3860f10b076-0.orc} which is not previously added.
Manifest might be corrupted.
{code}
was:In AbstractTableWrite, the Writer is cleaned up as soon as no new files
are generated, which may lead to the changes generated after the compaction
have not been committed, but the new data from the next checkpoint comes to
create a new writer, which conflicts with the changes generated in the next
round of checkpoint and the previous round, resulting in an exception.
> Clear RecordWriter slower to avoid causing frequent compaction conflicts
> ------------------------------------------------------------------------
>
> Key: FLINK-29295
> URL: https://issues.apache.org/jira/browse/FLINK-29295
> Project: Flink
> Issue Type: Bug
> Components: Table Store
> Reporter: Jingsong Lee
> Assignee: Jingsong Lee
> Priority: Major
> Fix For: table-store-0.3.0, table-store-0.2.1
>
>
> In AbstractTableWrite, the Writer is cleaned up as soon as no new files are
> generated, which may lead to the changes generated after the compaction have
> not been committed, but the new data from the next checkpoint comes to create
> a new writer, which conflicts with the changes generated in the next round of
> checkpoint and the previous round, resulting in an exception:
> {code:java}
> Caused by: java.lang.IllegalStateException: Trying to delete file
> {org.apache.flink.table.data.binary.BinaryRowData@5759f99e, 0, 0,
> data-7bf2498e-d0a1-42a5-97b7-b3860f10b076-0.orc} which is not previously
> added. Manifest might be corrupted.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
