tsreaper commented on code in PR #562: URL: https://github.com/apache/flink-table-store/pull/562#discussion_r1119793489
########## docs/content/docs/features/append-only-table.md: ########## @@ -0,0 +1,98 @@ +--- +title: "Append Only Table" +weight: 2 +type: docs +aliases: +- /features/append-only-table.html +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Append Only Table + +By specifying `'write-mode' = 'append-only'` when creating the table, user creates an append-only table. + +You can only insert a whole record into the table. No delete or update is supported and you cannot define primary keys. +This type of table is suitable for use cases that do not require updates (such as log data synchronization). + +## Distribution + +You also need to define bucket number for Append-only table, see [Bucket]({{< ref "docs/concepts/basic-concepts#bucket" >}}). + +It is recommended that you set the `bucket-key` field. Otherwise, the data will be hashed according to the whole row, +and the performance will be poor. + Review Comment: Create a "Performance" section which contains two sub-sections: "Bucketing" and "Compaction". ########## docs/content/docs/features/append-only-table.md: ########## @@ -0,0 +1,98 @@ +--- +title: "Append Only Table" +weight: 2 +type: docs +aliases: +- /features/append-only-table.html +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Append Only Table + +By specifying `'write-mode' = 'append-only'` when creating the table, user creates an append-only table. + +You can only insert a whole record into the table. No delete or update is supported and you cannot define primary keys. Review Comment: a whole record -> a complete record ########## docs/content/docs/features/append-only-table.md: ########## @@ -0,0 +1,98 @@ +--- +title: "Append Only Table" +weight: 2 +type: docs +aliases: +- /features/append-only-table.html +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Append Only Table + +By specifying `'write-mode' = 'append-only'` when creating the table, user creates an append-only table. + +You can only insert a whole record into the table. No delete or update is supported and you cannot define primary keys. +This type of table is suitable for use cases that do not require updates (such as log data synchronization). + +## Distribution + +You also need to define bucket number for Append-only table, see [Bucket]({{< ref "docs/concepts/basic-concepts#bucket" >}}). + +It is recommended that you set the `bucket-key` field. Otherwise, the data will be hashed according to the whole row, +and the performance will be poor. + +## Read order + +During stream reading, the order in each bucket is consumed according to the input order. You can use this append-only +table as a message queue. + +During full incremental stream reading, full read is read in the order of partition creation time. If your scene +partition creation may be out of order, you can configure `scan.plan-sort-partition` to read data after sorting +by partition fields to ensure the order. + +## Compaction + +By default, the sink node will automatically perform compaction to solve the small file problem. The following options +control the strategy of compaction: + +<table class="configuration table table-bordered"> + <thead> + <tr> + <th class="text-left" style="width: 20%">Key</th> + <th class="text-left" style="width: 15%">Default</th> + <th class="text-left" style="width: 10%">Type</th> + <th class="text-left" style="width: 55%">Description</th> + </tr> + </thead> + <tbody> + <tr> + <td><h5>write-only</h5></td> + <td style="word-wrap: break-word;">false</td> + <td>Boolean</td> + <td>If set to true, compactions and snapshot expiration will be skipped. This option is used along with dedicated compact jobs.</td> + </tr> + <tr> + <td><h5>compaction.min.file-num</h5></td> + <td style="word-wrap: break-word;">5</td> + <td>Integer</td> + <td>For file set [f_0,...,f_N], the minimum file number which satisfies sum(size(f_i)) >= targetFileSize to trigger a compaction for append-only table. This value avoids almost-full-file to be compacted, which is not cost-effective.</td> + </tr> + <tr> + <td><h5>compaction.early-max.file-num</h5></td> + <td style="word-wrap: break-word;">50</td> + <td>Integer</td> + <td>For file set [f_0,...,f_N], the maximum file number to trigger a compaction for append-only table, even if sum(size(f_i)) < targetFileSize. This value avoids pending too much small files, which slows down the performance.</td> + </tr> + </tbody> +</table> + +## Example + +```sql +CREATE TABLE MyTable ( + product_id BIGINT, + price DOUBLE, + sales BIGINT +) WITH ( + 'write-mode' = 'append-only', + 'bucket' = '8', + 'bucket-key' = 'product_id' +); +``` Review Comment: Move example to the top. What most users need is just an example. Fewer of them need information about performance. ########## docs/content/docs/features/primary-key-table.md: ########## @@ -158,12 +154,21 @@ Full compaction changelog producer can produce complete changelog for any type o {{< /hint >}} -## Changelog Tables without Primary Keys +## Sequence Field Review Comment: Change this as a sub-section of "Merge Engines", because it only affects the result of deduplicate merge engine. ########## docs/content/docs/features/append-only-table.md: ########## @@ -0,0 +1,98 @@ +--- +title: "Append Only Table" +weight: 2 +type: docs +aliases: +- /features/append-only-table.html +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Append Only Table + +By specifying `'write-mode' = 'append-only'` when creating the table, user creates an append-only table. + +You can only insert a whole record into the table. No delete or update is supported and you cannot define primary keys. +This type of table is suitable for use cases that do not require updates (such as log data synchronization). + +## Distribution + +You also need to define bucket number for Append-only table, see [Bucket]({{< ref "docs/concepts/basic-concepts#bucket" >}}). + +It is recommended that you set the `bucket-key` field. Otherwise, the data will be hashed according to the whole row, +and the performance will be poor. + +## Read order + +During stream reading, the order in each bucket is consumed according to the input order. You can use this append-only +table as a message queue. + +During full incremental stream reading, full read is read in the order of partition creation time. If your scene +partition creation may be out of order, you can configure `scan.plan-sort-partition` to read data after sorting +by partition fields to ensure the order. Review Comment: ```suggestion ## Streaming Read Order For streaming reads, records are produced in the following order: * For any two records from two different partitions * If `scan.plan-sort-partition` is set to true, the record with a smaller partition value will be produced first. * Otherwise, the record with an earlier partition creation time will be produced first. * For any two records from the same partition but two different buckets, the record with a smaller bucket order will be produced first. Bucket orders are determined by planners and are unchanged during the whole reading process. * For any two records from the same partition and the same bucket, the first written record will be produced first. ``` ########## docs/content/docs/features/primary-key-table.md: ########## @@ -158,12 +154,21 @@ Full compaction changelog producer can produce complete changelog for any type o {{< /hint >}} -## Changelog Tables without Primary Keys +## Sequence Field -Changelog tables can also be used without primary keys. Users can only insert or delete a whole record from the table. No update is supported. +By default, the primary key table determines the order according to the input order. However, in distributed computing, Review Comment: By default, primary key tables determine the latest record according to the input order. ########## docs/content/docs/features/primary-key-table.md: ########## @@ -158,12 +154,21 @@ Full compaction changelog producer can produce complete changelog for any type o {{< /hint >}} -## Changelog Tables without Primary Keys +## Sequence Field -Changelog tables can also be used without primary keys. Users can only insert or delete a whole record from the table. No update is supported. +By default, the primary key table determines the order according to the input order. However, in distributed computing, +there will be some cases that lead to data disorder. At this time, you can use a time field as `sequence.field`, so +the storage engine will use this field to sort to ensure the consistency of data sequence. For example: Review Comment: will use this field to determine the latest record instead. For example: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
