JingsongLi commented on code in PR #167:
URL: https://github.com/apache/flink-table-store/pull/167#discussion_r902557175
##########
docs/content/docs/development/write-table.md:
##########
@@ -192,3 +192,95 @@ There are three main places in the Table Store's sink
writer that take up memory
- The memory consumed by compaction for reading files, it can be adjusted by
the
`num-sorted-run.compaction-trigger` option to change the maximum number of
files to be merged.
- The memory consumed by writing file, which is not adjustable.
+
+
+## Scale Bucket
+
+Since the LSM trees are built against each bucket, the number of total buckets
dramatically influences the performance.
Review Comment:
We dont need to emphasize `LSM trees`, we have append only too.
##########
docs/content/docs/development/write-table.md:
##########
@@ -192,3 +192,95 @@ There are three main places in the Table Store's sink
writer that take up memory
- The memory consumed by compaction for reading files, it can be adjusted by
the
`num-sorted-run.compaction-trigger` option to change the maximum number of
files to be merged.
- The memory consumed by writing file, which is not adjustable.
+
+
+## Scale Bucket
+
+Since the LSM trees are built against each bucket, the number of total buckets
dramatically influences the performance.
+Table Store allows users to tune bucket numbers by `ALTER TABLE` command and
reorganize data layout by `INSERT OVERWRITE`
+without recreating the table/partition. When executing overwrite jobs, the
framework will automatically scan the data with
+the bucket number recorded in manifest file and hash the record according to
the current bucket numbers.
+
+#### Rescale Overwrite
+```sql
+-- scale number of total buckets
+ALTER TABLE table_dentifier SET ('bucket' = '...')
Review Comment:
table_dentifier -> table_identifier
##########
docs/content/docs/development/write-table.md:
##########
@@ -192,3 +192,95 @@ There are three main places in the Table Store's sink
writer that take up memory
- The memory consumed by compaction for reading files, it can be adjusted by
the
`num-sorted-run.compaction-trigger` option to change the maximum number of
files to be merged.
- The memory consumed by writing file, which is not adjustable.
+
+
+## Scale Bucket
+
+Since the LSM trees are built against each bucket, the number of total buckets
dramatically influences the performance.
+Table Store allows users to tune bucket numbers by `ALTER TABLE` command and
reorganize data layout by `INSERT OVERWRITE`
+without recreating the table/partition. When executing overwrite jobs, the
framework will automatically scan the data with
+the bucket number recorded in manifest file and hash the record according to
the current bucket numbers.
+
+#### Rescale Overwrite
+```sql
+-- scale number of total buckets
+ALTER TABLE table_dentifier SET ('bucket' = '...')
+
+-- reorganize data layout of table/partition
+INSERT OVERWRITE table_identifier [PARTITION (part_spec)]
+SELECT ...
+FROM table_identifier
+[WHERE part_spec]
+```
+
+Please beware that
Review Comment:
beware?
##########
docs/content/docs/development/write-table.md:
##########
@@ -192,3 +192,95 @@ There are three main places in the Table Store's sink
writer that take up memory
- The memory consumed by compaction for reading files, it can be adjusted by
the
`num-sorted-run.compaction-trigger` option to change the maximum number of
files to be merged.
- The memory consumed by writing file, which is not adjustable.
+
+
+## Scale Bucket
+
+Since the LSM trees are built against each bucket, the number of total buckets
dramatically influences the performance.
+Table Store allows users to tune bucket numbers by `ALTER TABLE` command and
reorganize data layout by `INSERT OVERWRITE`
+without recreating the table/partition. When executing overwrite jobs, the
framework will automatically scan the data with
+the bucket number recorded in manifest file and hash the record according to
the current bucket numbers.
+
+#### Rescale Overwrite
+```sql
+-- scale number of total buckets
+ALTER TABLE table_dentifier SET ('bucket' = '...')
+
+-- reorganize data layout of table/partition
+INSERT OVERWRITE table_identifier [PARTITION (part_spec)]
+SELECT ...
+FROM table_identifier
+[WHERE part_spec]
+```
+
+Please beware that
+- `ALTER TABLE` only modifies the table's metadata and will **NOT** reorganize
or reformat existing data.
+ Reorganize exiting data must be achieved by `INSERT OVERWRITE`.
+- Scale bucket number does not influence the read and running write jobs.
+- Once the bucket number is changed, any new `INSERT INTO` jobs without
reorganize table/partition
+ will throw a `TableException` with message like
+ ```text
+ Try to write table/partition ... with a new bucket num ...,
+ but the previous bucket num is ... Please switch to batch mode,
+ and perform INSERT OVERWRITE to rescale current data layout first.
+ ```
+
+
+{{< hint info >}}
+__Note:__ Currently, scale bucket is only supported for tables without
enabling log system.
+{{< /hint >}}
+
+#### Use Case
+
+Suppose there is a daily streaming ETL task to sync transaction data. The
table's DDL and pipeline
+are listed as follows.
+
+```sql
+-- table DDL
+CREATE TABLE verified_orders (
+ trade_order_id BIGINT,
+ item_id BIGINT,
+ item_price DOUBLE
+ dt STRING
+ PRIMARY KEY (dt, trade_order_id, item_id) NOT ENFORCED
+) PARTITIONED BY (dt)
+WITH (
+ 'bucket' = '16'
+);
+
+-- streaming insert as bucket num = 16
+INSERT INTO verified_orders
+SELECT trade_order_id,
+ item_id,
+ item_price,
+ DATE_FORMAT(gmt_create, 'yyyy-MM-dd') AS dt
+FROM raw_orders
+WHERE order_status = 'verified'
+```
+The pipeline has been running well for the past four weeks. However, the data
volume has grown fast recently,
+and the job's latency keeps increasing. A possible workaround is to create a
new table with a larger bucket number
+(thus the parallelism can be increased accordingly) and sync data to this new
table.
+
+However, there is a better solution with four steps.
+
+- First, suspend the streaming job.
+
+- Increase the bucket number.
+ ```sql
+ -- scaling out bucket number
+ ALTER TABLE verified_orders SET ('bucket' = '32')
+ ```
+
+- Switch to batch mode and `INSER OVERWRITE` the partition.
+ ```sql
+ -- reorganize the data layout as bucket num = 32
+ INSERT OVERWRITE verified_orders PARTITION (dt = 'yyyy-MM-dd')
+ SELECT trade_order_id,
+ item_id,
+ item_price
+ FROM verified_orders
+ WHERE dt = 'yyyy-MM-dd' AND order_status = 'verified'
+ ```
+
+- Recompile the streaming job and restore from the latest checkpoint.
Review Comment:
We can say: please use savepoint
##########
docs/content/docs/development/write-table.md:
##########
@@ -192,3 +192,95 @@ There are three main places in the Table Store's sink
writer that take up memory
- The memory consumed by compaction for reading files, it can be adjusted by
the
`num-sorted-run.compaction-trigger` option to change the maximum number of
files to be merged.
- The memory consumed by writing file, which is not adjustable.
+
+
+## Scale Bucket
+
+Since the LSM trees are built against each bucket, the number of total buckets
dramatically influences the performance.
+Table Store allows users to tune bucket numbers by `ALTER TABLE` command and
reorganize data layout by `INSERT OVERWRITE`
+without recreating the table/partition. When executing overwrite jobs, the
framework will automatically scan the data with
+the bucket number recorded in manifest file and hash the record according to
the current bucket numbers.
+
+#### Rescale Overwrite
+```sql
+-- scale number of total buckets
+ALTER TABLE table_dentifier SET ('bucket' = '...')
+
+-- reorganize data layout of table/partition
+INSERT OVERWRITE table_identifier [PARTITION (part_spec)]
+SELECT ...
+FROM table_identifier
+[WHERE part_spec]
+```
+
+Please beware that
+- `ALTER TABLE` only modifies the table's metadata and will **NOT** reorganize
or reformat existing data.
+ Reorganize exiting data must be achieved by `INSERT OVERWRITE`.
+- Scale bucket number does not influence the read and running write jobs.
+- Once the bucket number is changed, any new `INSERT INTO` jobs without
reorganize table/partition
Review Comment:
You can say that reading is perfectly fine as long as this partition does
not continue to write data.
##########
docs/content/docs/development/write-table.md:
##########
@@ -192,3 +192,95 @@ There are three main places in the Table Store's sink
writer that take up memory
- The memory consumed by compaction for reading files, it can be adjusted by
the
`num-sorted-run.compaction-trigger` option to change the maximum number of
files to be merged.
- The memory consumed by writing file, which is not adjustable.
+
+
+## Scale Bucket
+
+Since the LSM trees are built against each bucket, the number of total buckets
dramatically influences the performance.
+Table Store allows users to tune bucket numbers by `ALTER TABLE` command and
reorganize data layout by `INSERT OVERWRITE`
+without recreating the table/partition. When executing overwrite jobs, the
framework will automatically scan the data with
+the bucket number recorded in manifest file and hash the record according to
the current bucket numbers.
+
+#### Rescale Overwrite
+```sql
+-- scale number of total buckets
+ALTER TABLE table_dentifier SET ('bucket' = '...')
+
+-- reorganize data layout of table/partition
+INSERT OVERWRITE table_identifier [PARTITION (part_spec)]
+SELECT ...
+FROM table_identifier
+[WHERE part_spec]
+```
+
+Please beware that
+- `ALTER TABLE` only modifies the table's metadata and will **NOT** reorganize
or reformat existing data.
+ Reorganize exiting data must be achieved by `INSERT OVERWRITE`.
+- Scale bucket number does not influence the read and running write jobs.
+- Once the bucket number is changed, any new `INSERT INTO` jobs without
reorganize table/partition
+ will throw a `TableException` with message like
+ ```text
+ Try to write table/partition ... with a new bucket num ...,
+ but the previous bucket num is ... Please switch to batch mode,
+ and perform INSERT OVERWRITE to rescale current data layout first.
+ ```
+
+
+{{< hint info >}}
+__Note:__ Currently, scale bucket is only supported for tables without
enabling log system.
Review Comment:
I thought about it, currently the kafka topic is managed by the user, we
seem to only need to remind the user to modify the number of kafka topic
partitions can be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
