JingsongLi commented on code in PR #601: URL: https://github.com/apache/flink-web/pull/601#discussion_r1069583303
########## _posts/2023-01-15-release-table-store-0.3.0.md: ########## @@ -0,0 +1,238 @@ +--- +layout: post +title: "Apache Flink Table Store 0.3.0 Release Announcement" +date: 2023-01-15T08:00:00.000Z +categories: news +authors: +- JingsongLi: + name: "Jingsong Lee" + +excerpt: The Apache Flink Community is pleased to announce the release for Flink Table Store 0.3.0! + +--- + +The Apache Flink community is pleased to announce the release of the +[Apache Flink Table Store](https://github.com/apache/flink-table-store) (0.3.0). + +We highly recommend all users upgrade to Flink Table Store 0.3.0. 0.3.0 completed 150+ issues, which were completed by nearly 30 contributors. + +Please check out the full [documentation]({{site.DOCS_BASE_URL}}flink-table-store-docs-release-0.3/) for detailed information and user guides. + +<br/> + +Flink Table Store 0.3 completes many exciting features, Enhance its ability as a data lake and greatly +improve the availability of stream pipeline. Some important features are described below. + +## Changelog Producer: Full-Compaction + +If: +- You are using `partial-update` or `aggregation` table, at the time of writing, table store can't know what the + result is after merging, so table store can't generate the corresponding changelog. +- Your input can’t produce a complete changelog but you still want to get rid of the costly normalized operator, + +You may consider using the [Full compaction changelog producer]({{site.DOCS_BASE_URL}}flink-table-store-docs-release-0.3/docs/features/table-types/#full-compaction). + +By specifying `'changelog-producer' = 'full-compaction'`, Table Store will compare the results between full compactions +and produce the differences as changelog. The latency of changelog is affected by the frequency of full compactions. +By specifying `changelog-producer.compaction-interval` table property (default value 30min), users can define the +maximum interval between two full compactions to ensure latency. + +<center> +<img src="{{site.baseurl}}/img/blog/table-store/changelog-producer-full-compaction.png" width="100%"/> +</center> + +<br/> + +## Dedicated Compaction Job && Multiple Writers + +By default, Table Store writers will perform compaction as needed when writing records. This is sufficient for most use cases, but there are two downsides: + +* This may result in unstable write throughput because throughput might temporarily drop when performing a compaction. +* Compaction will mark some data files as "deleted". If multiple writers mark the same file a conflict will occur when + committing the changes. Table Store will automatically resolve the conflict, but this may result in job restarts. + +To avoid these downsides, users can also choose to skip compactions in writers, and run a +[dedicated job only for compaction]({{site.DOCS_BASE_URL}}flink-table-store-docs-release-0.3/docs/maintenance/write-performance/#dedicated-compaction-job). +As compactions are performed only by the dedicated job, writers can continuously write records without pausing and no conflicts will ever occur. + +To skip compactions in writers, set `write-only` to `true`. + +To run a dedicated job for compaction, follow these instructions. + +Flink SQL currently does not support statements related to compactions, so we have to submit the compaction job through `flink run`. + +Run the following command to submit a compaction job for the table. + +```bash +<FLINK_HOME>/bin/flink run \ + -c org.apache.flink.table.store.connector.action.FlinkActions \ + /path/to/flink-table-store-dist-{{< version >}}.jar \ + compact \ + --warehouse <warehouse-path> \ + --database <database-name> \ + --table <table-name> +``` + +## Aggregation Table + +Sometimes users only care about aggregated results. The +[aggregation merge engine]({{site.DOCS_BASE_URL}}flink-table-store-docs-release-0.3/docs/features/table-types/#aggregation) +aggregates each value field with the latest data one by one under the same primary key according to the aggregate function. + +Each field not part of the primary keys must be given an aggregate function, specified by the +`fields.<field-name>.aggregate-function` table property. + +For example: + +``` +CREATE TABLE MyTable ( + product_id BIGINT, + price DOUBLE, + sales BIGINT, + PRIMARY KEY (product_id) NOT ENFORCED +) WITH ( + 'merge-engine' = 'aggregation', + 'fields.price.aggregate-function' = 'max', + 'fields.sales.aggregate-function' = 'sum' +); +``` + +## Schema Evolution + +In version 0.2, the research and development of Schema Evolution has begun. In version 0.3, some of the capabilities +of Schema Evolution have been completed. You can use below operations in Spark-SQL (Flink SQL completes the following syntax in 1.17): + +- Adding New Columns +- Renaming Column Name +- Dropping Columns + +Flink Table Store ensures that the above operations are safe, and the old data will automatically adapt to +the new schema when it is read. + +For example: + +``` +CREATE TABLE T (i INT, j INT); + +INSERT INTO T (1, 1); + +ALTER TABLE T RENAME COLUMN i to a; Review Comment: I will add `ADD` and `DROP` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
