This is an automated email from the ASF dual-hosted git repository.
lzljs3620320 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-paimon.git
The following commit(s) were added to refs/heads/master by this push:
new 3bf2a8abf [doc] Merge file-operations and manage files into understand
files
3bf2a8abf is described below
commit 3bf2a8abf16cbf075fb76b73c0385f7343ac731c
Author: Jingsong <[email protected]>
AuthorDate: Thu Oct 12 17:50:52 2023 +0800
[doc] Merge file-operations and manage files into understand files
---
docs/content/concepts/append-only-table.md | 2 +-
docs/content/maintenance/configurations.md | 2 +-
docs/content/maintenance/manage-files.md | 110 -----------
.../understand-files.md} | 217 +++++++++++++++------
4 files changed, 154 insertions(+), 177 deletions(-)
diff --git a/docs/content/concepts/append-only-table.md
b/docs/content/concepts/append-only-table.md
index 98075de80..0a73e78ad 100644
--- a/docs/content/concepts/append-only-table.md
+++ b/docs/content/concepts/append-only-table.md
@@ -249,7 +249,7 @@ You can trigger action by shell script:
{{< generated/sort-compact >}}
The sort parallelism is the same as the sink parallelism, you can dynamically
specify it by add conf --table-conf sink.parallelism=<value>.
-Other config is the same as [Compact Table]({{< ref
"concepts/file-operations#compact-table" >}})
+Other config is the same as [Compact Table]({{< ref
"maintenance/dedicated-compaction" >}})
### Streaming Source
diff --git a/docs/content/maintenance/configurations.md
b/docs/content/maintenance/configurations.md
index c88676345..36f01a385 100644
--- a/docs/content/maintenance/configurations.md
+++ b/docs/content/maintenance/configurations.md
@@ -1,6 +1,6 @@
---
title: "Configurations"
-weight: 10
+weight: 100
type: docs
aliases:
- /maintenance/configurations.html
diff --git a/docs/content/maintenance/manage-files.md
b/docs/content/maintenance/manage-files.md
deleted file mode 100644
index ab4f8b21f..000000000
--- a/docs/content/maintenance/manage-files.md
+++ /dev/null
@@ -1,110 +0,0 @@
----
-title: "Manage Files"
-weight: 6
-type: docs
-aliases:
-- /maintenance/manage-files.html
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# Manage Small Files
-
-Many users are concerned about small files, which can lead to:
-1. Stability issue: Too many small files in HDFS, NameNode will be
overstressed.
-2. Cost issue: A small file in HDFS will temporarily use the size of a minimum
of one Block, for example 128 MB.
-3. Query efficiency: The efficiency of querying too many small files will be
affected.
-
-## Understand Checkpoints
-
-Assuming you are using Flink Writer, each checkpoint generates 1-2 snapshots,
and the checkpoint forces the files to be
-generated on DFS, so the smaller the checkpoint interval the more small files
will be generated.
-
-1. So first thing is increase checkpoint interval.
-
-By default, not only checkpoint will cause the file to be generated, but
writer's memory (write-buffer-size) exhaustion
-will also flush data to DFS and generate the corresponding file. You can
enable `write-buffer-spillable` to generate
-spilled files in writer to generate bigger files in DFS.
-
-2. So second thing is increase `write-buffer-size` or enable
`write-buffer-spillable`.
-
-## Understand Snapshots
-
-Before delving further into this section, please ensure that you have read
[File Operations]({{< ref "concepts/file-operations" >}}).
-
-{{< img src="/img/file-operations-3.png">}}
-
-Paimon maintains multiple versions of files, compaction and deletion of files
are logical and do not actually
-delete files. Files are only really deleted when Snapshot is expired, so the
first way to reduce files is to
-reduce the time it takes for snapshot to be expired. Flink writer will
automatically expire snapshots.
-
-See [Expire Snapshots]({{< ref
"/maintenance/manage-snapshots#expire-snapshots" >}}).
-
-## Understand Partitions and Buckets
-
-Paimon files are organized in a layered style. The following image illustrates
the file layout. Starting
-from a snapshot file, Paimon readers can recursively access all records from
the table.
-
-{{< img src="/img/file-layout.png">}}
-
-For example, the following table:
-
-```sql
-CREATE TABLE MyTable (
- user_id BIGINT,
- item_id BIGINT,
- behavior STRING,
- dt STRING,
- hh STRING,
- PRIMARY KEY (dt, hh, user_id) NOT ENFORCED
-) PARTITIONED BY (dt, hh) WITH (
- 'bucket' = '10'
-);
-```
-
-The table data will be physically sliced into different partitions, and
different buckets inside, so if the overall
-data volume is too small, there is at least one file in a single bucket, I
suggest you configure a smaller number
-of buckets, otherwise there will be quite a few small files as well.
-
-## Understand LSM for Primary Table
-
-LSM tree organizes files into several sorted runs. A sorted run consists of
one or multiple
-[data files]({{< ref "concepts/file-layouts#data-files" >}}) and each data
file belongs to exactly one sorted run.
-
-{{< img src="/img/sorted-runs.png">}}
-
-By default, sorted runs number depends on `num-sorted-run.compaction-trigger`,
see [Compaction for Primary Key Table]({{< ref
"/maintenance/write-performance#compaction" >}}),
-this means that there are at least 5 files in a bucket. If you want to reduce
this number, you can keep fewer files, but write performance may suffer.
-
-## Understand Files for Append-Only Table
-
-By default, Append-Only also does automatic compaction to reduce the number of
small files.
-
-However, for Bucket's Append-only table, it will only compact the files within
the Bucket for sequential
-purposes, which may keep more small files. See [Compaction for Append-Only
Table]({{< ref "/concepts/append-only-table#compaction" >}}).
-
-## Understand Full-Compaction
-
-Maybe you think the 5 files for the primary key table are actually okay, but
the Append-Only table (bucket)
-may have 50 small files in a single bucket, which is very difficult to accept.
Worse still, partitions that
-are no longer active also keep so many small files.
-
-It is recommended that you configure [Full-Compaction]({{< ref
"/maintenance/read-performance#full-compaction" >}}),
-configure ‘full-compaction.delta-commits’ perform full-compaction periodically
in Flink writing. And it can ensure
-that partitions are full compacted before writing ends.
diff --git a/docs/content/concepts/file-operations.md
b/docs/content/maintenance/understand-files.md
similarity index 67%
rename from docs/content/concepts/file-operations.md
rename to docs/content/maintenance/understand-files.md
index 588dae283..ceb2712e7 100644
--- a/docs/content/concepts/file-operations.md
+++ b/docs/content/maintenance/understand-files.md
@@ -1,9 +1,9 @@
---
-title: "File Operations"
-weight: 4
+title: "Understand Files"
+weight: 99
type: docs
aliases:
-- /concepts/file-operations.html
+- /maintenance/understand-files.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -24,14 +24,14 @@ specific language governing permissions and limitations
under the License.
-->
-# File Operations
+# Understand Files
-This article is specifically designed to clarify
-the impact that various file operations have on files.
+This article is specifically designed to clarify
+the impact that various file operations have on files.
-This page provides concrete examples and practical tips for
-effectively managing them. Furthermore, through an in-depth
-exploration of operations such as commit and compact,
+This page provides concrete examples and practical tips for
+effectively managing them. Furthermore, through an in-depth
+exploration of operations such as commit and compact,
we aim to offer insights into the creation and updates of files.
## Prerequisite
@@ -39,14 +39,16 @@ we aim to offer insights into the creation and updates of
files.
Before delving further into this page, please ensure that you have read
through the
following sections:
-1. [Basic Concepts]({{< ref "concepts/basic-concepts" >}}),
-2. [File Layouts]({{< ref "concepts/file-layouts" >}}) and
+1. [Basic Concepts]({{< ref "concepts/basic-concepts" >}}),
+2. [File Layouts]({{< ref "concepts/file-layouts" >}}) and
3. How to use Paimon in [Flink]({{< ref "engines/flink" >}}).
-## Create Catalog
+## Understand File Operations
-Start Flink SQL client via `./sql-client.sh` and execute the following
-statements one by one to create a Paimon catalog.
+### Create Catalog
+
+Start Flink SQL client via `./sql-client.sh` and execute the following
+statements one by one to create a Paimon catalog.
```sql
CREATE CATALOG paimon WITH (
'type' = 'paimon',
@@ -58,7 +60,7 @@ USE CATALOG paimon;
This will only create a directory at given path `file:///tmp/paimon`.
-## Create Table
+### Create Table
Execute the following create table statement will create a Paimon table with 3
fields:
@@ -72,11 +74,11 @@ CREATE TABLE T (
) PARTITIONED BY (dt);
```
-This will create Paimon table `T` under the path `/tmp/paimon/default.db/T`,
-with its schema stored in `/tmp/paimon/default.db/T/schema/schema-0`
+This will create Paimon table `T` under the path `/tmp/paimon/default.db/T`,
+with its schema stored in `/tmp/paimon/default.db/T/schema/schema-0`
-## Insert Records Into Table
+### Insert Records Into Table
Run the following insert statement in Flink SQL:
@@ -85,8 +87,8 @@ INSERT INTO T VALUES (1, 10001, 'varchar00001', '20230501');
```
Once the Flink job is completed, the records are written to the Paimon table
through a successful `commit`.
-Users can verify the visibility of these records by executing the query
`SELECT * FROM T` which will return a single row.
-The commit process creates a snapshot located at the path
`/tmp/paimon/default.db/T/snapshot/snapshot-1`.
+Users can verify the visibility of these records by executing the query
`SELECT * FROM T` which will return a single row.
+The commit process creates a snapshot located at the path
`/tmp/paimon/default.db/T/snapshot/snapshot-1`.
The resulting file layout at snapshot-1 is as described below:
{{< img src="/img/file-operations-0.png">}}
@@ -112,9 +114,9 @@ The content of snapshot-1 contains metadata of the
snapshot, such as manifest li
}
```
-Remind that a manifest list contains all changes of the snapshot,
`baseManifestList` is the base
-file upon which the changes in `deltaManifestList` is applied.
-The first commit will result in 1 manifest file, and 2 manifest lists are
+Remind that a manifest list contains all changes of the snapshot,
`baseManifestList` is the base
+file upon which the changes in `deltaManifestList` is applied.
+The first commit will result in 1 manifest file, and 2 manifest lists are
created (the file names might differ from those in your experiment):
```bash
@@ -123,19 +125,19 @@ manifest-list-4ccc-c07f-4090-958c-cfe3ce3889e5-1
manifest-list-4ccc-c07f-4090-958c-cfe3ce3889e5-0
manifest-2b833ea4-d7dc-4de0-ae0d-ad76eced75cc-0
```
-`manifest-2b833ea4-d7dc-4de0-ae0d-ad76eced75cc-0` is the manifest
+`manifest-2b833ea4-d7dc-4de0-ae0d-ad76eced75cc-0` is the manifest
file (manifest-1-0 in the above graph), which stores the information about the
data files in the snapshot.
-`manifest-list-4ccc-c07f-4090-958c-cfe3ce3889e5-0` is the
+`manifest-list-4ccc-c07f-4090-958c-cfe3ce3889e5-0` is the
baseManifestList (manifest-list-1-base in the above graph), which is
effectively empty.
-`manifest-list-4ccc-c07f-4090-958c-cfe3ce3889e5-1` is the
-deltaManifestList (manifest-list-1-delta in the above graph), which
-contains a list of manifest entries that perform operations on data
+`manifest-list-4ccc-c07f-4090-958c-cfe3ce3889e5-1` is the
+deltaManifestList (manifest-list-1-delta in the above graph), which
+contains a list of manifest entries that perform operations on data
files, which, in this case, is `manifest-1-0`.
-Now let's insert a batch of records across different partitions and
+Now let's insert a batch of records across different partitions and
see what happens. In Flink SQL, execute the following statement:
```sql
@@ -151,8 +153,8 @@ INSERT INTO T VALUES
(10, 10010, 'varchar00010', '20230510');
```
-The second `commit` takes place and executing `SELECT * FROM T` will return
-10 rows. A new snapshot, namely `snapshot-2`, is created and gives us the
+The second `commit` takes place and executing `SELECT * FROM T` will return
+10 rows. A new snapshot, namely `snapshot-2`, is created and gives us the
following physical file layout:
```bash
% ls -1tR .
@@ -199,17 +201,17 @@ schema-0
The new file layout as of snapshot-2 looks like
{{< img src="/img/file-operations-1.png">}}
-## Delete Records From Table
+### Delete Records From Table
-Now let's delete records that meet the condition `dt>=20230503`.
+Now let's delete records that meet the condition `dt>=20230503`.
In Flink SQL, execute the following statement:
{{< label Batch >}}
```sql
DELETE FROM T WHERE dt >= '20230503';
```
-The third `commit` takes place and it gives us `snapshot-3`. Now, listing the
files
-under the table and your will find out no partition is dropped. Instead, a new
data
+The third `commit` takes place and it gives us `snapshot-3`. Now, listing the
files
+under the table and your will find out no partition is dropped. Instead, a new
data
file is created for partition `20230503` to `20230510`:
```bash
@@ -218,9 +220,9 @@ data-b93f468c-b56f-4a93-adc4-b250b3aa3462-0.orc # newer
data file created by the
data-0fcacc70-a0cb-4976-8c88-73e92769a762-0.orc # older data file created by
the insert statement
```
-This make sense since we insert a record in the second commit (represented by
+This make sense since we insert a record in the second commit (represented by
`+I[10, 10010, 'varchar00010', '20230510']`) and then delete
-the record in the third commit. Executing `SELECT * FROM T` will return 2
rows, namely:
+the record in the third commit. Executing `SELECT * FROM T` will return 2
rows, namely:
```
+I[1, 10001, 'varchar00001', '20230501']
+I[2, 10002, 'varchar00002', '20230502']
@@ -229,12 +231,12 @@ the record in the third commit. Executing `SELECT * FROM
T` will return 2 rows,
The new file layout as of snapshot-3 looks like
{{< img src="/img/file-operations-2.png">}}
-Note that `manifest-3-0` contains 8 manifest entries of `ADD` operation type,
-corresponding to 8 newly written data files.
+Note that `manifest-3-0` contains 8 manifest entries of `ADD` operation type,
+corresponding to 8 newly written data files.
-## Compact Table
+### Compact Table
As you may have noticed, the number of small files will augment over successive
snapshots, which may lead to decreased read performance. Therefore, a
full-compaction
@@ -265,7 +267,7 @@ an example would be (suppose you're already in Flink home)
--path file:///tmp/paimon/default.db/T
```
-All current table files will be compacted and a new snapshot, namely
`snapshot-4`, is
+All current table files will be compacted and a new snapshot, namely
`snapshot-4`, is
made and contains the following information:
```json
@@ -291,13 +293,13 @@ made and contains the following information:
The new file layout as of snapshot-4 looks like
{{< img src="/img/file-operations-3.png">}}
-Note that `manifest-4-0` contains 20 manifest entries (18 `DELETE` operations
and 2 `ADD` operations)
+Note that `manifest-4-0` contains 20 manifest entries (18 `DELETE` operations
and 2 `ADD` operations)
1. For partition `20230503` to `20230510`, two `DELETE` operations for two
data files
-2. For partition `20230501` to `20230502`, one `DELETE` operation and one
`ADD` operation
+2. For partition `20230501` to `20230502`, one `DELETE` operation and one
`ADD` operation
for the same data file.
-## Alter Table
+### Alter Table
Execute the following statement to configure full-compaction:
```sql
ALTER TABLE T SET ('full-compaction.delta-commits' = '1');
@@ -306,54 +308,54 @@ ALTER TABLE T SET ('full-compaction.delta-commits' = '1');
It will create a new schema for Paimon table, namely `schema-1`, but no
snapshot
has actually used this schema yet until the next commit.
-## Expire Snapshots
+### Expire Snapshots
-Remind that the marked data files are not truly deleted until the snapshot
expires and
+Remind that the marked data files are not truly deleted until the snapshot
expires and
no consumer depends on the snapshot. For more information, see [Expiring
Snapshots]({{< ref "maintenance/manage-snapshots#expiring-snapshots" >}}).
-During the process of snapshot expiration, the range of snapshots is initially
determined, and then data files within these snapshots are marked for deletion.
-A data file is `marked` for deletion only when there is a manifest entry of
kind `DELETE` that references that specific data file.
+During the process of snapshot expiration, the range of snapshots is initially
determined, and then data files within these snapshots are marked for deletion.
+A data file is `marked` for deletion only when there is a manifest entry of
kind `DELETE` that references that specific data file.
This marking ensures that the file will not be utilized by subsequent
snapshots and can be safely removed.
Let's say all 4 snapshots in the above diagram are about to expire. The expire
process is as follows:
-1. It first deletes all marked data files, and records any changed buckets.
-
-2. It then deletes any changelog files and associated manifests.
-
+1. It first deletes all marked data files, and records any changed buckets.
+
+2. It then deletes any changelog files and associated manifests.
+
3. Finally, it deletes the snapshots themselves and writes the earliest hint
file.
If any directories are left empty after the deletion process, they will be
deleted as well.
Let's say another snapshot, `snapshot-5` is created and snapshot expiration is
triggered. `snapshot-1` to `snapshot-4` are
-to be deleted. For simplicity, we will only focus on files from previous
snapshots, the final layout after snapshot
+to be deleted. For simplicity, we will only focus on files from previous
snapshots, the final layout after snapshot
expiration looks like:
{{< img src="/img/file-operations-4.png">}}
As a result, partition `20230503` to `20230510` are physically deleted.
-## Flink Stream Write
+### Flink Stream Write
Finally, we will examine Flink Stream Write by utilizing the example
-of CDC ingestion. This section will address the capturing and writing of
-change data into Paimon, as well as the mechanisms behind asynchronous compact
+of CDC ingestion. This section will address the capturing and writing of
+change data into Paimon, as well as the mechanisms behind asynchronous compact
and snapshot commit and expiration.
-To begin, let's take a closer look at the CDC ingestion workflow and
+To begin, let's take a closer look at the CDC ingestion workflow and
the unique roles played by each component involved.
{{< img src="/img/cdc-ingestion-topology.png">}}
-1. `MySQL CDC Source` uniformly reads snapshot and incremental data, with
`SnapshotReader` reading snapshot data
+1. `MySQL CDC Source` uniformly reads snapshot and incremental data, with
`SnapshotReader` reading snapshot data
and `BinlogReader` reading incremental data, respectively.
2. `Paimon Sink` writes data into Paimon table in bucket level. The
`CompactManager` within it will trigger compaction
asynchronously.
3. `Committer Operator` is a singleton responsible for committing and expiring
snapshots.
-Next, we will go over end-to-end data flow.
+Next, we will go over end-to-end data flow.
{{< img src="/img/cdc-ingestion-source.png">}}
@@ -363,9 +365,9 @@ Next, we will go over end-to-end data flow.
{{< img src="/img/cdc-ingestion-write.png">}}
-`Paimon Sink` first buffers new records in a heap-based LSM tree, and flushes
them to disk when
+`Paimon Sink` first buffers new records in a heap-based LSM tree, and flushes
them to disk when
the memory buffer is full. Note that each data file written is a sorted run.
At this point, no manifest file and snapshot
-is created. Right before Flink checkpoint takes places, `Paimon Sink` will
flush all buffered records and send committable message
+is created. Right before Flink checkpoint takes places, `Paimon Sink` will
flush all buffered records and send committable message
to downstream, which is read and committed by `Committer Operator` during
checkpoint.
{{< img src="/img/cdc-ingestion-commit.png">}}
@@ -375,9 +377,94 @@ contains information about all data files in the table.
{{< img src="/img/cdc-ingestion-compact.png">}}
-At later point asynchronous compaction might take place, and the committable
produced by `CompactManager` contains information
-about previous files and merged files so that `Committer Operator` can
construct corresponding manifest entries. In this case
-`Committer Operator` might produce two snapshot during Flink checkpoint, one
for data written (snapshot of kind `Append`) and the
-other for compact (snapshot of kind `Compact`). If no data file is written
during checkpoint interval, only snapshot of kind `Compact`
+At later point asynchronous compaction might take place, and the committable
produced by `CompactManager` contains information
+about previous files and merged files so that `Committer Operator` can
construct corresponding manifest entries. In this case
+`Committer Operator` might produce two snapshot during Flink checkpoint, one
for data written (snapshot of kind `Append`) and the
+other for compact (snapshot of kind `Compact`). If no data file is written
during checkpoint interval, only snapshot of kind `Compact`
will be created. `Committer Operator` will check against snapshot expiration
and perform
physical deletion of marked data files.
+
+## Understand Small Files
+
+Many users are concerned about small files, which can lead to:
+1. Stability issue: Too many small files in HDFS, NameNode will be
overstressed.
+2. Cost issue: A small file in HDFS will temporarily use the size of a minimum
of one Block, for example 128 MB.
+3. Query efficiency: The efficiency of querying too many small files will be
affected.
+
+### Understand Checkpoints
+
+Assuming you are using Flink Writer, each checkpoint generates 1-2 snapshots,
and the checkpoint forces the files to be
+generated on DFS, so the smaller the checkpoint interval the more small files
will be generated.
+
+1. So first thing is increase checkpoint interval.
+
+By default, not only checkpoint will cause the file to be generated, but
writer's memory (write-buffer-size) exhaustion
+will also flush data to DFS and generate the corresponding file. You can
enable `write-buffer-spillable` to generate
+spilled files in writer to generate bigger files in DFS.
+
+2. So second thing is increase `write-buffer-size` or enable
`write-buffer-spillable`.
+
+### Understand Snapshots
+
+Before delving further into this section, please ensure that you have read
[File Layouts]({{< ref "concepts/file-layouts" >}}).
+
+{{< img src="/img/file-operations-3.png">}}
+
+Paimon maintains multiple versions of files, compaction and deletion of files
are logical and do not actually
+delete files. Files are only really deleted when Snapshot is expired, so the
first way to reduce files is to
+reduce the time it takes for snapshot to be expired. Flink writer will
automatically expire snapshots.
+
+See [Expire Snapshots]({{< ref
"/maintenance/manage-snapshots#expire-snapshots" >}}).
+
+### Understand Partitions and Buckets
+
+Paimon files are organized in a layered style. The following image illustrates
the file layout. Starting
+from a snapshot file, Paimon readers can recursively access all records from
the table.
+
+{{< img src="/img/file-layout.png">}}
+
+For example, the following table:
+
+```sql
+CREATE TABLE MyTable (
+ user_id BIGINT,
+ item_id BIGINT,
+ behavior STRING,
+ dt STRING,
+ hh STRING,
+ PRIMARY KEY (dt, hh, user_id) NOT ENFORCED
+) PARTITIONED BY (dt, hh) WITH (
+ 'bucket' = '10'
+);
+```
+
+The table data will be physically sliced into different partitions, and
different buckets inside, so if the overall
+data volume is too small, there is at least one file in a single bucket, I
suggest you configure a smaller number
+of buckets, otherwise there will be quite a few small files as well.
+
+### Understand LSM for Primary Table
+
+LSM tree organizes files into several sorted runs. A sorted run consists of
one or multiple
+[data files]({{< ref "concepts/file-layouts#data-files" >}}) and each data
file belongs to exactly one sorted run.
+
+{{< img src="/img/sorted-runs.png">}}
+
+By default, sorted runs number depends on `num-sorted-run.compaction-trigger`,
see [Compaction for Primary Key Table]({{< ref
"/maintenance/write-performance#compaction" >}}),
+this means that there are at least 5 files in a bucket. If you want to reduce
this number, you can keep fewer files, but write performance may suffer.
+
+### Understand Files for Append-Only Table
+
+By default, Append-Only also does automatic compaction to reduce the number of
small files.
+
+However, for Bucket's Append-only table, it will only compact the files within
the Bucket for sequential
+purposes, which may keep more small files. See [Compaction for Append-Only
Table]({{< ref "/concepts/append-only-table#compaction" >}}).
+
+### Understand Full-Compaction
+
+Maybe you think the 5 files for the primary key table are actually okay, but
the Append-Only table (bucket)
+may have 50 small files in a single bucket, which is very difficult to accept.
Worse still, partitions that
+are no longer active also keep so many small files.
+
+It is recommended that you configure [Full-Compaction]({{< ref
"/maintenance/read-performance#full-compaction" >}}),
+configure ‘full-compaction.delta-commits’ perform full-compaction periodically
in Flink writing. And it can ensure
+that partitions are full compacted before writing ends.
