This is an automated email from the ASF dual-hosted git repository.
lzljs3620320 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-paimon.git
The following commit(s) were added to refs/heads/master by this push:
new 09782ee40 [doc] Document file compression/format per level
09782ee40 is described below
commit 09782ee402f25eae28d2f896a82c3a628c92e7a2
Author: Jingsong <[email protected]>
AuthorDate: Mon Jul 24 21:39:45 2023 +0800
[doc] Document file compression/format per level
---
docs/content/maintenance/write-performance.md | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/docs/content/maintenance/write-performance.md
b/docs/content/maintenance/write-performance.md
index 2c9cd5744..21ac5aa6a 100644
--- a/docs/content/maintenance/write-performance.md
+++ b/docs/content/maintenance/write-performance.md
@@ -159,6 +159,18 @@ metadata.stats-mode = none
The collection of statistical information for row storage is a bit expensive,
so I suggest turning off statistical
information as well.
+If you don't want to modify all files to Avro format, at least you can
consider modifying the files in the previous
+layers to Avro format. You can use `'file.format.per.level' = '0:avro,1:avro'`
to specify the files in the first two
+layers to be in Avro format.
+
+## File Compression
+
+By default, Paimon uses high-performance compression algorithms such as LZ4
and SNAPPY. But their compression rate
+will be not so good. If you can reduce the write/read performance, you can
modify the compression algorithm:
+
+1. `'file.compression'`: Default file compression format. If you need a higher
compression rate, I recommend using `'ZSTD'`.
+2. `'file.compression.per.level'`: Define different compression policies for
different level. For example `'0:lz4,1:zstd'`.
+
## Stability
If there are too few buckets, or too few resources, full-compaction may cause
checkpoint to timeout, Flink's default
