[
https://issues.apache.org/jira/browse/FLINK-21891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-21891:
----------------------------
Component/s: Connectors / Hive
> The .staging_xxx directory isn't deleted after writing data to hive table in
> batch mode
> ---------------------------------------------------------------------------------------
>
> Key: FLINK-21891
> URL: https://issues.apache.org/jira/browse/FLINK-21891
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Table SQL / Runtime
> Affects Versions: 1.12.0
> Reporter: Re0 Beatrice
> Priority: Major
>
> In flink 1.12.0, use Blink Planner to read data from Hbase and write the
> results to Hive via Flink SQL.
> The .staging_xxx files on HDFS:
> /user/hive/warehouse/user.db/t_hive_user_group_result/.staging_1616074732697
> /user/hive/warehouse/user.db/t_hive_user_group_result/.staging_1616120408195
> /user/hive/warehouse/user.db/t_hive_user_group_result/.staging_1616121007337
> /user/hive/warehouse/user.db/t_hive_user_group_result/.staging_1616121607484
> /user/hive/warehouse/user.db/t_hive_user_group_result/.staging_1616206808142
> I found the following code in
> `org.apache.flink.table.filesystem.FileSystemOutputFormat` caused the problem:
> {code:java}
> import java.io.File;
> @Override
> public void finalizeGlobal(int parallelism) {
> try {
> FileSystemCommitter committer = new FileSystemCommitter(
> fsFactory, msFactory, overwrite, tmpPath, partitionColumns.length);
>
> committer.commitUpToCheckpoint(CHECKPOINT_ID);
> } catch (Exception e) {
> throw new TableException("Exception in finalizeGlobal", e);
> } finally {
> new File(tmpPath.getPath()).delete(); // the error code
> }
> }
> {code}
> The code in finally code block `new File(..)` can't convert `tmpPath` to HDFS
> file instance, I think the following code is more correct and works for me:
> {code:java}
> fsFactory.create(tmpPath.toUri()).delete(tmpPath, true);
> {code}
> A similar code has appeared in the class of PartitionTempFileManager.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
