[
https://issues.apache.org/jira/browse/FLINK-23918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450830#comment-17450830
]
luoyuxia edited comment on FLINK-23918 at 11/30/21, 8:52 AM:
-------------------------------------------------------------
[~liwei_snake] Thanks for reporting it.
For the first question:
I think it may caused by external factors about hdfs such like namenode swith
(if you have) or the file is deleted by someone others or some jobs.
For the second question:
It's not expected to compact one file twice. Does it always happen or just
happen after failover?
Anyway, I will try to find the cause.
Or maybe you can share your insight.
was (Author: luoyuxia):
[~liwei_snake] Thanks for reporting it.
Have you ever changed the parallelism for the job?
If not, I think it may caused by external factors about hdfs such like namenode
swith (if you have) or the file is deleted by someone others or some jobs.
Or maybe you can share your insight.
> FileNotFoundException when open auto-compact in filesystem
> ----------------------------------------------------------
>
> Key: FLINK-23918
> URL: https://issues.apache.org/jira/browse/FLINK-23918
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Ecosystem
> Affects Versions: 1.12.2
> Environment: Flink 1.12.2
> HDFS 3.2.1
> config:
> flink parallelism is 8
> task manager slot is 4
> task manager memory is 8g
> checkpoint min interval is 60s
> state backend use: rocksdb
> state ttl: 3 days
> Reporter: li wei
> Priority: Blocker
> Attachments: image-2021-08-23-18-09-17-674.png,
> image-2021-08-23-18-14-20-389.png, image-2021-08-23-18-14-35-275.png
>
>
> After I config auto-compaction=true in filesystem connector, sometimes it
> will restart and show me exception like this "Caused by:
> java.io.FileNotFoundException: File does not exist:
> hdfs://xxxxxx/tmp/xxxxx/dt=20210823/.uncompacted-part-ef59d89c-36fb-49c5-9b74-dd6a7882ea9f-0-241".
> And when I use this filename as keyword to search, I found that it first show
> me an exception, then I got 2 compact success logs.
> I think it's not right here, First, flink should not throw this exception;
> Second, this file should not be merged into 2 files.
> !image-2021-08-23-18-14-20-389.png!
> !image-2021-08-23-18-14-35-275.png!
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
