luoyuxia commented on code in PR #1612:
URL: https://github.com/apache/fluss/pull/1612#discussion_r2309230336
##########
fluss-common/src/main/java/org/apache/fluss/lake/writer/WriterInitContext.java:
##########
@@ -58,16 +56,9 @@ public interface WriterInitContext {
String partition();
/**
- * Returns the table schema.
- *
- * @return the table schema
- */
- Schema schema();
-
- /**
- * Returns the table custom properties.
+ * Returns the table info.
Review Comment:
nit:
```suggestion
* Returns the Fluss table info.
```
##########
website/docs/maintenance/operations/upgrading.md:
##########
@@ -83,4 +83,7 @@ The compatibility between the Fluss client and the Fluss
server is described in
| | Server 0.6 | Server 0.7 | Limitations |
|------------|------------|------------|-------------|
| Client 0.6 | ✔️ | ✔️ | |
-| Client 0.7 | ✔️ | ✔️ | |
\ No newline at end of file
+| Client 0.7 | ✔️ | ✔️ | |
+
+## Compatibility between Fluss datalake tiering service
Review Comment:
```suggestion
## Upgrading Fluss datalake tiering service
```
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/tiering/PaimonLakeWriter.java:
##########
@@ -43,7 +48,8 @@ public PaimonLakeWriter(
PaimonCatalogProvider paimonCatalogProvider, WriterInitContext
writerInitContext)
throws IOException {
this.paimonCatalog = paimonCatalogProvider.get();
- FileStoreTable fileStoreTable =
getTable(writerInitContext.tablePath());
+ FileStoreTable fileStoreTable =
+ getTable(writerInitContext.tablePath(),
writerInitContext.tableInfo());
Review Comment:
add `getDataLakeAutoCompaction` in `TableConfig` so that pass
`writerInitContext.tableInfo().getDataLakeAutoCompaction` to getTable method.
##########
fluss-lake/fluss-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/tiering/PaimonTieringTest.java:
##########
@@ -661,15 +708,8 @@ public String partition() {
}
@Override
- public Map<String, String> customProperties() {
- // don't care about table custom properties for Paimon
lake writer
- return new HashMap<>();
- }
-
- @Override
- public org.apache.fluss.metadata.Schema schema() {
- throw new UnsupportedOperationException(
- "The lake writer in Paimon currently uses
paimonCatalog to determine the schema.");
+ public TableInfo tableInfo() {
Review Comment:
Is it possible to use
```
// we don't really care about the table info in this test,
// so return a fake tableInfo to simplify the test
return TableInfo.of(
tablePath, 0, 0,
TableDescriptor.builder().build(), 0L, 0L);
```
?
##########
website/docs/maintenance/operations/upgrading.md:
##########
@@ -83,4 +83,7 @@ The compatibility between the Fluss client and the Fluss
server is described in
| | Server 0.6 | Server 0.7 | Limitations |
|------------|------------|------------|-------------|
| Client 0.6 | ✔️ | ✔️ | |
-| Client 0.7 | ✔️ | ✔️ | |
\ No newline at end of file
+| Client 0.7 | ✔️ | ✔️ | |
+
+## Compatibility between Fluss datalake tiering service
Review Comment:
I draft a more detail doc for the changes, currently it's not very clear:
```
## Upgrading Fluss datalake tiering service
### Behavior Change
Since Fluss 0.8, the **auto-compaction feature during datalake tiering is
now disabled by default**. Compaction will no longer be triggered automatically
as part of the tiering process.
| Version | Behavior
|
|----------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|
| **Previous Version(v0.7 and earlier)** | Auto-compaction enabled during
tiering
|
| **Fluss 0.8+** | **Auto-compaction disabled by
default**. Tiering service focus solely on data movement; compaction must be
explicitly enabled. |
### How to Enable Compaction
To maintain the previous behavior and enable automatic compaction, you must
manually configure `table.datalake.auto-compaction = true` for each table in
table option.
**Important Note**: This is a per-table setting. This design provides
granular control, allowing you to manage compaction based on the specific
performance and storage needs of each individual table.
### Reason for Change & Benefits
This change was implemented to significantly improve the core stability and
performance of the Datalake Tiering Service.
- **Enhanced Stability & Performance**: Compaction is a resource-intensive
operation (CPU/I/O) that can impose significant pressure on the tiering
service. By disabling it by default, the service can dedicate all resources to
its primary function: reliably and efficiently moving data. This results in a
more stable, predictable, and smoother tiering experience for all users.
- **Granular Control & Flexibility**: This change empowers users to make a
conscious choice based on their specific needs. You can now decide the optimal
balance between storage efficiency (achieved through compaction) and
computational resource allocation on a per-table basis.
```
##########
fluss-common/src/main/java/org/apache/fluss/lake/writer/WriterInitContext.java:
##########
@@ -18,14 +18,12 @@
package org.apache.fluss.lake.writer;
import org.apache.fluss.annotation.PublicEvolving;
-import org.apache.fluss.metadata.Schema;
import org.apache.fluss.metadata.TableBucket;
+import org.apache.fluss.metadata.TableInfo;
import org.apache.fluss.metadata.TablePath;
import javax.annotation.Nullable;
-import java.util.Map;
-
/**
* The WriterInitContext interface provides the context needed to create a
LakeWriter. It includes
* methods to obtain the table path, table bucket, and an optional partition.
Review Comment:
nit: also update this java doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@fluss.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
