This is an automated email from the ASF dual-hosted git repository.
zhangshenghang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/seatunnel-website.git
The following commit(s) were added to refs/heads/main by this push:
new 3f1801c0d438 [Docs] Move user cases content into blog (#473)
3f1801c0d438 is described below
commit 3f1801c0d438321067dcee66ee5b28bea8daa0b6
Author: David Zollo <[email protected]>
AuthorDate: Wed Jul 1 21:45:04 2026 +0800
[Docs] Move user cases content into blog (#473)
Co-authored-by: Daniel <[email protected]>
---
README.md | 1 -
README_ZH.md | 1 -
{user_cases => blog}/2022-2-18-Meetup-vip.md | 196 ++++++++++-----------
docusaurus.config.js | 33 ++--
.../2022-2-18-Meetup-vip.md | 196 ++++++++++-----------
i18n/zh-CN/docusaurus-theme-classic/navbar.json | 6 +-
6 files changed, 220 insertions(+), 213 deletions(-)
diff --git a/README.md b/README.md
index 0256bedd7d40..9d9090642262 100644
--- a/README.md
+++ b/README.md
@@ -47,7 +47,6 @@ This website is compiled using node, using Docusaurus
framework components
├── README_ZH.md
├── babel.config.js
├── blog
-├── user_cases
├── community // Community
├── docusaurus.config.js
|-- download // Download
diff --git a/README_ZH.md b/README_ZH.md
index baceace7f3af..115dddb1b5a3 100644
--- a/README_ZH.md
+++ b/README_ZH.md
@@ -47,7 +47,6 @@ asf-staging 官网测试环境 通过https://seatunnel.staged.apache.org 访问
├── README_ZH.md
├── babel.config.js
├── blog
-├── user_cases
├── community // 社区
├── docusaurus.config.js
|-- download // 下载
diff --git a/user_cases/2022-2-18-Meetup-vip.md b/blog/2022-2-18-Meetup-vip.md
similarity index 99%
rename from user_cases/2022-2-18-Meetup-vip.md
rename to blog/2022-2-18-Meetup-vip.md
index 5b1386c49ccf..9e2b571afc2a 100644
--- a/user_cases/2022-2-18-Meetup-vip.md
+++ b/blog/2022-2-18-Meetup-vip.md
@@ -1,98 +1,98 @@
----
-slug: The-practice-of-SeaTunnel-in-Vip
-title: The practice of SeaTunnel in Vip
-tags: [Vip, ClickHouse]
----
-
-Guest speaker: Vip Senior Big Data Engineer Wang Yu
-Lecture preparation: Zhang Detong
-
-Introduction: Vip referenced SeaTunnel as early as version 1.0. We use
SeaTunnel to perform some data interaction work between Hive and ClickHouse.
-Today's presentation will focus on the following points:
-
-* Requirements and pain points of ClickHouse data import;
-* Selection of ClickHouse warehousing and warehousing tools;
-* Hive to ClickHouse;
-* ClickHouse to Hive;
-* Integration of SeaTunnel and Vipshop data platform;
-* Future outlook;
-
-# Requirements and pain points of ClickHouse data import
-## 1. Vipshop Data OLAP Architecture
-The picture shows the OLAP architecture of Vipshop. The modules we are
responsible for are the data service and the computing engine in the picture.
The underlying data warehouses are divided into offline data warehouses,
real-time data warehouses, and lake warehouses. For computing engines, we use
Presto, Kylin and Clickhouse. Although Clickhouse is a storage-integrated OLAP
database, we have included it in the computing engine part in order to take
advantage of Clickhouse's excellent c [...]
-
-
-## 2. Requirements
-We connect the underlying Hive, Kudu, and Alluxio components through Presto
Connector and Spark components. Big data components can import and export data
to and from each other, and you can use appropriate components to analyze data
according to the needs and scenarios of data analysis. But when we introduced
Clickhouse, it was a data island, and it was difficult to import and export
data. There is a lot of work between Hive and Clickhouse to implement import
and export. Our first data [...]
-
-
-The second requirement is that Presto runs SQL relatively slowly. The figure
shows an example of slow SQL. The SQL where condition in the figure sets the
date, time range, and specific filter conditions. This kind of SQL usage runs
slowly because Presto uses partition granularity to push down. Even after
optimization by other methods such as Hive's bucket table and bucketing, the
return time is a few seconds, which cannot meet business requirements. In this
case, we need to use Clickhous [...]
-
-
-Our real-time data is written to Clickhouse through Kafka and Flink SQL.
However, it is not enough to use real-time data for analysis. It is necessary
to use the Hive dimension table and the T+1 real-time table with the ETL
calculation number for accelerated transportation in Clickhouse. This requires
importing Hive data into Clickhouse, which is our third requirement.
-
-
-## 3. Pain points
-First, we introduce a data component to consider its performance. The
granularity of the Hive table is five minutes. Is there a component that can
support a short ETL process and import the ETL results into Clickhouse within
five minutes? Second, we need to ensure the quality of the data, and the
accuracy of the data needs to be guaranteed. The number of data entries in Hive
and Clickhouse needs to be consistent. If there is a problem with the data
quality, can the data be repaired by re [...]
-
-# Selection of ClickHouse and Hive warehousing and warehousing tools
-Based on the pain points in the data business, we have compared and selected
data warehouse and warehouse tools. We mainly choose among open source tools,
without considering commercial warehouse entry and exit tools, we mainly
compare DataX, SeaTunnel, and write Spark programs and use jdbc to insert
ClickHouse among the three options.
-SeaTunnel and Spark rely on Vipshop's own Yarn cluster, which can directly
implement distributed reading and writing. DataX is non-distributed, and the
startup process between Reader and Writer takes a long time, and the
performance is ordinary. The performance of SeaTunnel and Spark for data
processing can reach several times that of DataX.
-Data of more than one billion can run smoothly in SeaTunnel and Spark. DataX
has great performance pressure after the amount of data is large, and it is
difficult to process data of more than one billion.
-In terms of read and write plug-in scalability, SeaTunnel supports a variety
of data sources and supports users to develop plug-ins. SeaTunnel supports data
import into Redis.
-In terms of stability, since SeaTunnel and DataX are self-contained tools, the
stability will be better. The stability aspect of Spark requires attention to
code quality.
-
-
-The amount of data in our exposure table is in the billions of levels every
day. We have the performance requirement to complete data processing within 5
minutes. We have the need to import and export data to Redis. We need import
and export tools that can be connected to the data platform for task
scheduling. . For the consideration of data volume, performance, scalability,
and platform compatibility, we chose SeaTunnel as our data warehouse import and
export tool.
-# Import Hive data into ClickHouse
-The following will introduce our use of SeaTunnel.
-The picture is a Hive table, which is our three-level product dimension table,
including category products, dimension categories, and user population
information. The primary key of the table is a third-level category
ct_third_id, and the following value is the bitmap of two uids, which is the
bitmap type of the user id. We need to import this Hive table into Clickhouse.
-
-
-SeaTunnel is easy to install, and the official website documentation describes
how to install it. The figure below shows the configuration of SeaTunnel. In
the configuration, env, source and sink are essential. In the env part, the
example in the figure is the Spark configuration. The configuration includes
concurrency, etc. These parameters can be adjusted. The source part is the data
source. The Hive data source is configured here, including a Hive Select
statement. Spark runs the SQL [...]
-
-
-Run SeaTunnel, execute the sh script file, configure the conf file address and
yarn information, and then you can.
-
-Spark logs are generated during the running process, and both successful
running and running errors can be viewed in the logs.
-
-
-In order to better fit the business, Vipshop will make some improvements to
SeaTunnel. All our ETL tasks need to be rerun. We support pre_sql and check_sql
to implement data rerun and logarithm. The main process is to execute pre_sql
for preprocessing after the data is ready, delete the old partition data in
Clickhouse, store it in a directory, and restore the partition and rename when
it fails. check_sql will check, and the whole process will end after the check
is passed; if the check [...]
-
-
-
-Based on the 1.0 version of SeaTunnel, Vipshop has added RowBinary for
acceleration, and it also makes it easier to import the binary files of
HuperLogLog and BinaryBitmap from Hive to Clickhouse. We made changes in
ClickHouse-jdbc, bulk_size, Hive-source. Use the extended api of CK-jdbc to
write data to CK in rowbinary mode. Bulk_size introduces the control logic for
writing to CK in rowbinary mode. Hive-source
-RDD is partitioned with HashPartitioner to break up data to prevent data from
being skewed.
-
-We also let SeaTunnel support multiple types. In order to circle the crowd,
corresponding methods need to be implemented in Clickhouse, Preso, and Spark.
We added Callback, HttpEntity, and RowBinaryStream that support Batch feature
to Clickhouse-jdbc, added bitmap type mapping to Clickhouse-jdbc and
Clickhouse-sink code, and implemented UDF of Clickhouse's Hyperloglog and
Bitmap functions in Presto and Spark.
-In the previous configuration, the clickhouse-sink part can specify the table
name, and here is the difference between writing to the local table and the
distributed table. The performance of writing to a distributed table is worse
than that of writing to a local table, which will put more pressure on the
Clickhouse cluster. However, in scenarios such as exposure meter, flow meter,
and ABTest, two tables are required to join, and both tables are in the order
of billions. . At this time, [...]
-For the writing of local tables and distributed tables, our future
transformation direction is to implement consistent hashing in Seatunnel, write
directly according to certain rules, such as Clickhouse, without relying on
Clickhouse itself for data distribution, and improve Clickhouse's high CPU load
problem.
-
-# ClickHouse data import into Hive
-We have the needs of people in the circle. Every day, Vipshop gathers 200,000
people in the supplier circle, such as people born in the 1980s, Gao Fushuai,
and Bai Fumei. These Bitmap crowd information in Clickhouse needs to be
exported to the Hive table, coordinated with other ETL tasks in Hive, and
finally pushed to PIKA for use by external media. We made SeaTunnel back-push
Clickhouse Bitmap crowd data to Hive.
-
-
-The figure shows the SeaTunnel configuration. We configure the source as
Clickhouse, the sink as Hive, and the data verification is also configured in
Hive.
-
-
-Since we access SeaTunnel earlier, we have processed some modules, including
adding plugin-spark-sink-hive module, plugin-spark-source-ClickHouse module,
and rewriting Spark Row related methods so that they can be packaged through
The Clickhouse data mapped by Schem, reconstruct the StructField and generate
the DataFrame that finally needs to land on Hive. The latest version has many
source and sink components, which is more convenient to use in SeaTunnel. It is
now also possible to inte [...]
-
-# Integration of SeaTunnel and Vipshop Data Platform
-Each company has its own scheduling system, such as Beluga, Zeus. The
scheduling tool of Vipshop is Shufang, and the scheduling tool integrates the
data transmission tool. The following is the architecture diagram of the
scheduling system, which includes the entry and exit of various types of data.
-
-
-The SeaTunnel task type is integrated into the platform. The picture is a
screenshot of the scheduled task of Shufang. You can see that the selected part
is a configured SeaTunnel task. resource information. The following shows the
historical running instance information.
-
-
-We integrated SeaTunnel into the scheduling system. The Shufang Scheduling
Master will assign tasks to the corresponding Agents according to the task
types, and assign them to the appropriate machines to run according to the
Agent load. The controller pulls the task scheduling configuration and
information in the foreground. After arriving, a SeaTunnel cluster is generated
and executed in a virtual environment similar to k8s pod and cgroup isolation.
The running result will be judged by [...]
-
-
-SeaTunnel itself is a tool-based component, which is used to manage and
control data blood relationship, data quality, historical records, high-alert
monitoring, and resource allocation. We integrate SeaTunnel into the platform,
and we can take advantage of the platform to take advantage of SeaTunnel.
-SeaTunnel is used for processing in the deposit crowd. By managing data, we
divide the circled people into different people according to their paths and
usage conditions, or thousands of people and thousands of faces, tag users, and
push a certain type of people circled to users, analysts and suppliers.
-
-
-The traffic enters Kafka, enters the warehouse through Flink, and then forms a
user label table through ETL. After the user label table is generated, we use
the BitMap method implemented by Presto to type the data into a wide table in
Hive. Users create tasks by box-selecting entries on the crowd system page,
submit to Tengqun, and generate SQL query Clickhouse BitMap. Clickhouse's
BitMap query speed is very fast. Due to its inherent advantages, we need to
import Hive's BitMap table into [...]
-During the whole process, SeaTunnel is responsible for exporting the data from
Hive to Clickhouse. After the ETL process of Clickhouse is completed, SeaTunnel
exports the data from Clickhouse to Hive.
-In order to fulfill this requirement, we implemented UDFs of ClickHouse's
Hyperloglog and BitMap functions on Presto and Spark; we also developed the
Seatunnel interface, so that the crowds circled by users using the Bitmap
method in ClickHouse can be directly written to the Hive table through
Seatunnel , without an intermediate landing step. Users can also call the
SeaTunnel interface through spark to circle the crowd or reverse the crowd
bitmap in Hive, so that the data can be directly [...]
-# Follow-up
-In the future, we will further improve the problem of high CPU load when
Clickhouse writes data. In the next step, we will implement the CK-local mode
of Clickhouse data source and read end in SeaTunnel, separate read and write,
and reduce Clickhouse pressure. In the future we will also add more sink
support, such as data push to Pika and corresponding data checking.
+---
+slug: The-practice-of-SeaTunnel-in-Vip
+title: The practice of SeaTunnel in Vip
+tags: [Vip, ClickHouse]
+---
+
+Guest speaker: Vip Senior Big Data Engineer Wang Yu
+Lecture preparation: Zhang Detong
+
+Introduction: Vip referenced SeaTunnel as early as version 1.0. We use
SeaTunnel to perform some data interaction work between Hive and ClickHouse.
+Today's presentation will focus on the following points:
+
+* Requirements and pain points of ClickHouse data import;
+* Selection of ClickHouse warehousing and warehousing tools;
+* Hive to ClickHouse;
+* ClickHouse to Hive;
+* Integration of SeaTunnel and Vipshop data platform;
+* Future outlook;
+
+# Requirements and pain points of ClickHouse data import
+## 1. Vipshop Data OLAP Architecture
+The picture shows the OLAP architecture of Vipshop. The modules we are
responsible for are the data service and the computing engine in the picture.
The underlying data warehouses are divided into offline data warehouses,
real-time data warehouses, and lake warehouses. For computing engines, we use
Presto, Kylin and Clickhouse. Although Clickhouse is a storage-integrated OLAP
database, we have included it in the computing engine part in order to take
advantage of Clickhouse's excellent c [...]
+
+
+## 2. Requirements
+We connect the underlying Hive, Kudu, and Alluxio components through Presto
Connector and Spark components. Big data components can import and export data
to and from each other, and you can use appropriate components to analyze data
according to the needs and scenarios of data analysis. But when we introduced
Clickhouse, it was a data island, and it was difficult to import and export
data. There is a lot of work between Hive and Clickhouse to implement import
and export. Our first data [...]
+
+
+The second requirement is that Presto runs SQL relatively slowly. The figure
shows an example of slow SQL. The SQL where condition in the figure sets the
date, time range, and specific filter conditions. This kind of SQL usage runs
slowly because Presto uses partition granularity to push down. Even after
optimization by other methods such as Hive's bucket table and bucketing, the
return time is a few seconds, which cannot meet business requirements. In this
case, we need to use Clickhous [...]
+
+
+Our real-time data is written to Clickhouse through Kafka and Flink SQL.
However, it is not enough to use real-time data for analysis. It is necessary
to use the Hive dimension table and the T+1 real-time table with the ETL
calculation number for accelerated transportation in Clickhouse. This requires
importing Hive data into Clickhouse, which is our third requirement.
+
+
+## 3. Pain points
+First, we introduce a data component to consider its performance. The
granularity of the Hive table is five minutes. Is there a component that can
support a short ETL process and import the ETL results into Clickhouse within
five minutes? Second, we need to ensure the quality of the data, and the
accuracy of the data needs to be guaranteed. The number of data entries in Hive
and Clickhouse needs to be consistent. If there is a problem with the data
quality, can the data be repaired by re [...]
+
+# Selection of ClickHouse and Hive warehousing and warehousing tools
+Based on the pain points in the data business, we have compared and selected
data warehouse and warehouse tools. We mainly choose among open source tools,
without considering commercial warehouse entry and exit tools, we mainly
compare DataX, SeaTunnel, and write Spark programs and use jdbc to insert
ClickHouse among the three options.
+SeaTunnel and Spark rely on Vipshop's own Yarn cluster, which can directly
implement distributed reading and writing. DataX is non-distributed, and the
startup process between Reader and Writer takes a long time, and the
performance is ordinary. The performance of SeaTunnel and Spark for data
processing can reach several times that of DataX.
+Data of more than one billion can run smoothly in SeaTunnel and Spark. DataX
has great performance pressure after the amount of data is large, and it is
difficult to process data of more than one billion.
+In terms of read and write plug-in scalability, SeaTunnel supports a variety
of data sources and supports users to develop plug-ins. SeaTunnel supports data
import into Redis.
+In terms of stability, since SeaTunnel and DataX are self-contained tools, the
stability will be better. The stability aspect of Spark requires attention to
code quality.
+
+
+The amount of data in our exposure table is in the billions of levels every
day. We have the performance requirement to complete data processing within 5
minutes. We have the need to import and export data to Redis. We need import
and export tools that can be connected to the data platform for task
scheduling. . For the consideration of data volume, performance, scalability,
and platform compatibility, we chose SeaTunnel as our data warehouse import and
export tool.
+# Import Hive data into ClickHouse
+The following will introduce our use of SeaTunnel.
+The picture is a Hive table, which is our three-level product dimension table,
including category products, dimension categories, and user population
information. The primary key of the table is a third-level category
ct_third_id, and the following value is the bitmap of two uids, which is the
bitmap type of the user id. We need to import this Hive table into Clickhouse.
+
+
+SeaTunnel is easy to install, and the official website documentation describes
how to install it. The figure below shows the configuration of SeaTunnel. In
the configuration, env, source and sink are essential. In the env part, the
example in the figure is the Spark configuration. The configuration includes
concurrency, etc. These parameters can be adjusted. The source part is the data
source. The Hive data source is configured here, including a Hive Select
statement. Spark runs the SQL [...]
+
+
+Run SeaTunnel, execute the sh script file, configure the conf file address and
yarn information, and then you can.
+
+Spark logs are generated during the running process, and both successful
running and running errors can be viewed in the logs.
+
+
+In order to better fit the business, Vipshop will make some improvements to
SeaTunnel. All our ETL tasks need to be rerun. We support pre_sql and check_sql
to implement data rerun and logarithm. The main process is to execute pre_sql
for preprocessing after the data is ready, delete the old partition data in
Clickhouse, store it in a directory, and restore the partition and rename when
it fails. check_sql will check, and the whole process will end after the check
is passed; if the check [...]
+
+
+
+Based on the 1.0 version of SeaTunnel, Vipshop has added RowBinary for
acceleration, and it also makes it easier to import the binary files of
HuperLogLog and BinaryBitmap from Hive to Clickhouse. We made changes in
ClickHouse-jdbc, bulk_size, Hive-source. Use the extended api of CK-jdbc to
write data to CK in rowbinary mode. Bulk_size introduces the control logic for
writing to CK in rowbinary mode. Hive-source
+RDD is partitioned with HashPartitioner to break up data to prevent data from
being skewed.
+
+We also let SeaTunnel support multiple types. In order to circle the crowd,
corresponding methods need to be implemented in Clickhouse, Preso, and Spark.
We added Callback, HttpEntity, and RowBinaryStream that support Batch feature
to Clickhouse-jdbc, added bitmap type mapping to Clickhouse-jdbc and
Clickhouse-sink code, and implemented UDF of Clickhouse's Hyperloglog and
Bitmap functions in Presto and Spark.
+In the previous configuration, the clickhouse-sink part can specify the table
name, and here is the difference between writing to the local table and the
distributed table. The performance of writing to a distributed table is worse
than that of writing to a local table, which will put more pressure on the
Clickhouse cluster. However, in scenarios such as exposure meter, flow meter,
and ABTest, two tables are required to join, and both tables are in the order
of billions. . At this time, [...]
+For the writing of local tables and distributed tables, our future
transformation direction is to implement consistent hashing in Seatunnel, write
directly according to certain rules, such as Clickhouse, without relying on
Clickhouse itself for data distribution, and improve Clickhouse's high CPU load
problem.
+
+# ClickHouse data import into Hive
+We have the needs of people in the circle. Every day, Vipshop gathers 200,000
people in the supplier circle, such as people born in the 1980s, Gao Fushuai,
and Bai Fumei. These Bitmap crowd information in Clickhouse needs to be
exported to the Hive table, coordinated with other ETL tasks in Hive, and
finally pushed to PIKA for use by external media. We made SeaTunnel back-push
Clickhouse Bitmap crowd data to Hive.
+
+
+The figure shows the SeaTunnel configuration. We configure the source as
Clickhouse, the sink as Hive, and the data verification is also configured in
Hive.
+
+
+Since we access SeaTunnel earlier, we have processed some modules, including
adding plugin-spark-sink-hive module, plugin-spark-source-ClickHouse module,
and rewriting Spark Row related methods so that they can be packaged through
The Clickhouse data mapped by Schem, reconstruct the StructField and generate
the DataFrame that finally needs to land on Hive. The latest version has many
source and sink components, which is more convenient to use in SeaTunnel. It is
now also possible to inte [...]
+
+# Integration of SeaTunnel and Vipshop Data Platform
+Each company has its own scheduling system, such as Beluga, Zeus. The
scheduling tool of Vipshop is Shufang, and the scheduling tool integrates the
data transmission tool. The following is the architecture diagram of the
scheduling system, which includes the entry and exit of various types of data.
+
+
+The SeaTunnel task type is integrated into the platform. The picture is a
screenshot of the scheduled task of Shufang. You can see that the selected part
is a configured SeaTunnel task. resource information. The following shows the
historical running instance information.
+
+
+We integrated SeaTunnel into the scheduling system. The Shufang Scheduling
Master will assign tasks to the corresponding Agents according to the task
types, and assign them to the appropriate machines to run according to the
Agent load. The controller pulls the task scheduling configuration and
information in the foreground. After arriving, a SeaTunnel cluster is generated
and executed in a virtual environment similar to k8s pod and cgroup isolation.
The running result will be judged by [...]
+
+
+SeaTunnel itself is a tool-based component, which is used to manage and
control data blood relationship, data quality, historical records, high-alert
monitoring, and resource allocation. We integrate SeaTunnel into the platform,
and we can take advantage of the platform to take advantage of SeaTunnel.
+SeaTunnel is used for processing in the deposit crowd. By managing data, we
divide the circled people into different people according to their paths and
usage conditions, or thousands of people and thousands of faces, tag users, and
push a certain type of people circled to users, analysts and suppliers.
+
+
+The traffic enters Kafka, enters the warehouse through Flink, and then forms a
user label table through ETL. After the user label table is generated, we use
the BitMap method implemented by Presto to type the data into a wide table in
Hive. Users create tasks by box-selecting entries on the crowd system page,
submit to Tengqun, and generate SQL query Clickhouse BitMap. Clickhouse's
BitMap query speed is very fast. Due to its inherent advantages, we need to
import Hive's BitMap table into [...]
+During the whole process, SeaTunnel is responsible for exporting the data from
Hive to Clickhouse. After the ETL process of Clickhouse is completed, SeaTunnel
exports the data from Clickhouse to Hive.
+In order to fulfill this requirement, we implemented UDFs of ClickHouse's
Hyperloglog and BitMap functions on Presto and Spark; we also developed the
Seatunnel interface, so that the crowds circled by users using the Bitmap
method in ClickHouse can be directly written to the Hive table through
Seatunnel , without an intermediate landing step. Users can also call the
SeaTunnel interface through spark to circle the crowd or reverse the crowd
bitmap in Hive, so that the data can be directly [...]
+# Follow-up
+In the future, we will further improve the problem of high CPU load when
Clickhouse writes data. In the next step, we will implement the CK-local mode
of Clickhouse data source and read end in SeaTunnel, separate read and write,
and reduce Clickhouse pressure. In the future we will also add more sink
support, such as data push to Pika and corresponding data checking.
diff --git a/docusaurus.config.js b/docusaurus.config.js
index ab8e4bbf6e94..359b9f5d2afc 100644
--- a/docusaurus.config.js
+++ b/docusaurus.config.js
@@ -281,12 +281,6 @@ const config = {
position: "right",
activeBaseRegex: `/blog`,
},
- // {
- // to: "/user_cases",
- // label: "UserCases",
- // position: "right",
- // activeBaseRegex: `/user_cases`,
- // },
{
to: "/team",
label: "Team",
@@ -451,11 +445,30 @@ const config = {
},
],
[
- "@docusaurus/plugin-content-blog",
+ "@docusaurus/plugin-client-redirects",
{
- id: "user_cases",
- path: "user_cases",
- routeBasePath: "user_cases",
+ redirects: [
+ {
+ from: "/user_cases",
+ to: "/blog",
+ },
+ {
+ from: "/user_cases/tags",
+ to: "/blog/tags",
+ },
+ {
+ from: "/user_cases/The-practice-of-SeaTunnel-in-Vip",
+ to: "/blog/The-practice-of-SeaTunnel-in-Vip",
+ },
+ {
+ from: "/user_cases/tags/vip",
+ to: "/blog/tags/vip",
+ },
+ {
+ from: "/user_cases/tags/click-house",
+ to: "/blog/tags/click-house",
+ },
+ ],
},
],
],
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-blog-user_cases/2022-2-18-Meetup-vip.md
b/i18n/zh-CN/docusaurus-plugin-content-blog/2022-2-18-Meetup-vip.md
similarity index 99%
rename from
i18n/zh-CN/docusaurus-plugin-content-blog-user_cases/2022-2-18-Meetup-vip.md
rename to i18n/zh-CN/docusaurus-plugin-content-blog/2022-2-18-Meetup-vip.md
index fc31d0e2d813..181d4c57a723 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-blog-user_cases/2022-2-18-Meetup-vip.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-blog/2022-2-18-Meetup-vip.md
@@ -1,98 +1,98 @@
----
-slug: SeaTunnel 在唯品会的实践
-title: SeaTunnel 在唯品会的实践
-tags: [唯品会, ClickHouse]
----
-
-分享嘉宾：唯品会 资深大数据工程师 王玉
-讲稿整理：张德通
-
-导读: 唯品会早在1.0版本时就引用了SeaTunnel，我们使用SeaTunnel进行一些Hive到ClickHouse之间数据交互的工作。
-今天的介绍会围绕下面几点展开：
-
-* ClickHouse数据导入的需求和痛点；
-* ClickHouse出仓入仓工具选型；
-* Hive to ClickHouse；
-* ClickHouse to Hive；
-* SeaTunnel与唯品会数据平台的集成；
-* 未来展望；
-
-# ClickHouse数据导入的需求和痛点
-## 1.唯品会数据OLAP架构
-图中是唯品会OLAP架构，我们负责的模块是图中的数据服务和计算引擎两大部分。底层依赖的数据仓库分为离线数仓、实时数仓和湖仓。计算引擎方面，我们使用Presto、Kylin和Clickhouse。虽然Clickhouse是一个存储一体的OLAP数据库，我们为了利用Clickhouse的优秀计算性能而将它归入了计算引擎部分。基于OLAP组件之上，我们提供了SQL类数据服务和非SQL的唯品会自主分析，为不同智能服务。例如非SQL服务是为BI和商务提供更贴近业务的数据分析的服务。在数据服务至上抽象了多个数据应用。
-
-
-## 2.需求
-我们通过Presto
Connector和Spark组件，把底层的Hive、Kudu、Alluxio组件打通。大数据组件之间可以互相导入导出数据，可以根据数据分析的需求和场景任意利用合适的组件分析数据。但我们引入Clickhouse时，它是一个数据孤岛，数据的导入和导出比较困难。Hive和Clickhouse之间需要做很多工作才能实现导入导出。我们的第一个数据导入导出需求就是提升导入导出效率，把Clickhouse纳入大数据体系中。
-
-
-第二个需求是Presto跑SQL比较慢，图中是一个慢SQL的例子。图中的SQL
where条件设置了日期、时间范围和具体过滤条件，这类SQL使用由于Presto使用分区粒度下推，运行比较慢。即使用Hive的Bucket表和分桶等其他方式优化后也是几秒的返回时间、不能满足业务要求。这种情况下，我们需要利用Clickhouse做离线的OLAP计算加速。
-
-
-我们的实时数据是通过Kafka、Flink
SQL方式写入到Clickhouse中。但分析时只用实时数据是不够的，需要用Hive维度表和已经ETL计算号的T+1实时表一起在Clickhouse中做加速运输。这需要把Hive的数据导入到Clickhouse中，这就是我们的第三个需求。
-
-
-## 3.痛点
-首先，我们引入一项数据组件时要考虑其性能。Hive表粒度是五分钟，是否有组件可以支撑五分钟内完成一个短小ETL流程并把ETL结果导入到Clickhouse中？第二，我们需要保证数据质量，数据的准确性需要有保障。Hive和Clickhouse的数据条数需要保障一致性，如果数据质量出问题能否通过重跑等机制修复数据？第三，数据导入需要支持的数据类型是否完备？不同数据库之间的数据类型和一些机制不同，我们有HiperLogLog和BitMap这类在某一存储引擎中利用得比较多得数据类型，是否可以正确传输和识别，且可以较好地使用。
-
-# ClickHouse和Hive出仓入仓工具的选型
-基于数据业务上的痛点，我们对数据出仓入仓工具进行了对比和选择。我们主要在开源工具中进行选择，没有考虑商业出入仓工具，主要对比DataX、SeaTunnel和编写Spark程序并用jdbc插入ClickHouse这三个方案中取舍。
-SeaTunnel和Spark依赖唯品会自己的Yarn集群，可以直接实现分布式读取和写入。DataX是非分布式的，且Reader、Writer之间的启动过程耗时时间长，性能普通，SeaTunnel和Spark处理数据的性能可以达到DataX的数倍。
-十亿以上的数据可以平稳地在SeaTunnel和Spark中运行，DataX在数据量大以后性能压力大，处理十亿以上数据吃力。
-在读写插件扩展性方面，SeaTunnel支持了多种数据源，支持用户开发插件。SeaTunnel支持了数据导入Redis。
-稳定性上，SeaTunnel和DataX由于是自成体系的工具，稳定性会更好。Spark的稳定性方面需要关注代码质量。
-
-
-我们的曝光表数据量每天在几十亿级，我们有5min内完成数据处理的性能要求，我们我们存在数据导入导出到Redis的需求，我们需要导入导出工具可以接入到数据平台上进行任务调度。
出于数据量级、性能、可扩展性、平台兼容性几方面的考虑，我们选择了SeaTunnel作为我们的数仓导入导出工具。
-# Hive数据导入到ClickHouse
-下面将介绍我们对SeaTunnel的使用。
-图中是一张Hive表，它是我们三级的商品维度表，包含品类商品、维度品类和用户人群信息。表的主键是一个三级品类ct_third_id，下面的value是两个uid的位图，是用户id的bitmap类型，我们要把这个Hive表导入到Clickhouse。
-
-
-SeaTunnel安装简单，官网文档有介绍如何安装。下图中是SeaTunnel的配置，配置中env、source和sink是必不可少的。env部分，图中的例子是Spark配置，配置了包括并发度等，可以调整这些参数。source部分是数据来源，这里配置了Hive数据源，包括一条Hive
Select语句，Spark运行source配置中的SQL把数据读出，此处支持UDF进行简单ETL；sink部分配置了Clickhouse，可以看到output_type=rowbinary，rowbinary是唯品会自研加速方案；pre_sql和check_sql是自研的用于数据校验的功能，后面也会详细介绍；clickhouse.socket_timeout和bulk_size都是可以根据实际情况进行调整的。
-
-
-运行SeaTunnel，执行sh脚本文件、配置conf文件地址和yarn信息，后即可。
-
-运行过程中会产生Spark日志，运行成功和运行中错误都可以在日志中查看。
-
-
-为了更贴合业务，唯品会对SeaTunnel做了一些改进。我们的ETL任务都是需要重跑的，我们支持了pre_sql和check_sql实现数据的重跑和对数。主要流程是在数据准备好后，执行pre_sql进行预处理，在Clickhouse中执行删除旧分区数据、存放到某一目录下在失败时恢复该分区、rename这类操作。check_sql会检验，校验通过后整个流程结束；如果检验不通过，根据配置进行重跑，重跑不通过则报警到对应负责人。
-
-
-
-唯品会基于1.0版本SeaTunnel增加了RowBinary做加速，也让HuperLogLog和BinaryBitmap的二进制文件能更容易地从Hive导入到Clickhouse。我们在ClickHouse-jdbc、bulk_size、Hive-source几处进行了修改。使用CK-jdbc的extended
api，以rowbinary方式将数据写入CK，bulk_size引入了以rowbinary方式写入CK的控制逻辑，Hive-source
-RDD以HashPartitioner进行分区将数据打散，防止数据倾斜。
-
-我们还让SeaTunnel支持了多类型，为了圈人群的功能，需要在Clickhouse、Preso、Spark中实现对应的方法。我们在Clickhouse-jdbc中增加支持Batch特性的Callback、HttpEntity、RowBinaryStream，在Clickhouse-jdbc和Clickhouse-sink代码中增加了bitmap类型映射，在Presto和Spark中实现了Clickhouse的Hyperloglog和Bitmap的function的UDF。
-前面的配置中，Clickhouse-sink部分可以指定表名，这里有写入本地表和分布式表的差异。写入分布式表的性能比写入本地表差对Clickhouse集群的压力会更大，但在计算曝光表、流量表，ABTest等场景中需要两表Join，两张表量级均在几十亿。这时我们希望Join
key落在本机，Join成本更小。我们建表时在Clickhouse的分布式表分布规则中配置murmurHash64规则，然后在Seatunnel的sink里直接配置分布式表，把写入规则交给Clickhouse，利用了分布式表的特性进行写入。写入本地表对Clickhouse的压力会更小，写入的性能也会更好。我们在Seatunnel里，根据sink的本地表，去Clickhouse的System.cluster表里获取表的分布信息和机器分布host。然后根据均分规则写入这些host。把数据分布式写入的事情放到Seatunnel里来做。
-针对本地表和分布式表的写入，我们未来的改造方向是在Seatunnel实现一致性哈希，直接按照一定规则写如Clickhouse、不依赖Clickhouse自身做数据分发，改善Clickhouse高CPU负载问题。
-
-# ClickHouse数据导入到Hive
-我们有圈人群的需求，每天唯品会为供应商圈20万个人群，比如80后、高富帅、白富美的人群集合。这些在Clickhouse中的Bitmap人群信息需要导出到Hive表，在Hive中与其他ETL任务进行配合，最后推到PIKA交给外部媒体使用。我们使SeaTunnel将Clickhouse
Bitmap人群数据反推到Hive。
-
-
-图中是SeaTunnel配置，我们把source配置为Clickhouse、sink配置为Hive，数据校验也配置在Hive内。
-
-
-由于我们接入SeaTunnel较早，我们对一些模块间进行了加工，包括新增plugin-spark-sink-hive模块、plugin-spark-source-ClickHouse模块，重写Spark
Row相关方法，使其能封装经过Schem映射后的Clickhouse数据，重新构造StructField并生成最终需要落地Hive的DataFrame。最新版本已经有了很多source和sink组件，在SeaTunnel使用上更方便。现在也可以在SeaTunnel中直接集成Flink
connector。
-
-# SeaTunnel与唯品会数据平台的集成
-各个公司都有自己的调度系统，例如白鲸、宙斯。唯品会的调度工具是数坊，调度工具中集成了数据传输工具。下面是调度系统架构图，其中包含各类数据的出入仓。
-
-
-SeaTunnel任务类型集成到平台中，图中是数坊的定时任务截图，可以看到选中的部分，是一个配置好的SeaTunnel任务，负责人、最近一次耗时，前后依赖任务的血缘信息，消耗的资源信息。下面展示了历史运行实例信息。
-
-
-我们把SeaTunnel集成到了调度系统中，数坊调度Master会根据任务类型把任务分配到对应的Agent上，根据Agent负载情况分配到合适的机器上运行，管控器把前台的任务调度配置和信息拉取到后生成SeaTunnel
cluster，在类似于k8s
pod、cgroup隔离的虚拟环境内进行执行。运行结果会由调度平台的数据质量监控判断任务是否完成、是否运行成功，失败时进行重跑和告警。
-
-
-SeaTunnel本身是一个工具化的组件，是为了进行数据血缘，数据质量，历史记录，高警监控，还包括资源分配这些信息的管控。我们把SeaTunnel集成到平台中，可以利用平台优势利用好SeaTunnel。
-圈存人群中利用了SeaTunnel进行处理。我们通过打点数据，把圈存人群按照路径和使用情况分为不同的人，或称千人千面，把用户打上标签，圈出的某一类人群推送给用户、分析师和供应商。
-
-
-流量进入Kafka，通过Flink入仓，再通过ETL形成用户标签表，用户标签表生成后，我们通过Presto实现了的BitMap方法，把数据打成Hive中的宽表。用户通过在人群系统页面中框选词条创建任务，提交腾群，生成SQL查询Clickhouse
BitMap。Clickhouse的BitMap查询速度非常快，由天生优势，我们需要把Hive的BitMap表通过SeaTunnel导入到Clickhouse中。圈完人群后我们需要把表落地，形成Clickhouse的一个分区或一条记录，再把生成的结果BitMap表通过SeaTunnel存储到Hive中去。最后同步工具会将Hive的BitMap人群结果同步给外部媒体仓库Pika。每天圈20w个人群左右。
-整个过程中SeaTunnel负责把数据从Hive导出到Clickhouse，Clickhouse的ETL流程完成后SeaTunnel把数据从Clickhouse导出到Hive。
-为了完成这样的需求，我们在Presto和Spark端现ClickHouse的Hyperloglog和BitMap的function的UDF；我们还开发Seatunnel接口，使得用户在ClickHouse里使用Bitmap方法圈出来的人群，可以直接通过Seatunnel写入Hive表，无需中间落地步骤。用户也可以在Hive里通过spark圈人群或者反解人群bitmap，调用SeaTunnel接口，使数据直接传输到ClickHouse的结果表，而无需中间落地。
-# 后续工作
-后续我们会进一步改善Clickhouse写入数据时CPU负载高的问题，下一步会在SeaTunnel中实现Clickhouse数据源和读取端的CK-local模式，读写分离，减轻Clickhouse压力。未来我们也会增加更多sink支持，如数据推送到Pika和相应的数据检查。
+---
+slug: SeaTunnel 在唯品会的实践
+title: SeaTunnel 在唯品会的实践
+tags: [唯品会, ClickHouse]
+---
+
+分享嘉宾：唯品会 资深大数据工程师 王玉
+讲稿整理：张德通
+
+导读: 唯品会早在1.0版本时就引用了SeaTunnel，我们使用SeaTunnel进行一些Hive到ClickHouse之间数据交互的工作。
+今天的介绍会围绕下面几点展开：
+
+* ClickHouse数据导入的需求和痛点；
+* ClickHouse出仓入仓工具选型；
+* Hive to ClickHouse；
+* ClickHouse to Hive；
+* SeaTunnel与唯品会数据平台的集成；
+* 未来展望；
+
+# ClickHouse数据导入的需求和痛点
+## 1.唯品会数据OLAP架构
+图中是唯品会OLAP架构，我们负责的模块是图中的数据服务和计算引擎两大部分。底层依赖的数据仓库分为离线数仓、实时数仓和湖仓。计算引擎方面，我们使用Presto、Kylin和Clickhouse。虽然Clickhouse是一个存储一体的OLAP数据库，我们为了利用Clickhouse的优秀计算性能而将它归入了计算引擎部分。基于OLAP组件之上，我们提供了SQL类数据服务和非SQL的唯品会自主分析，为不同智能服务。例如非SQL服务是为BI和商务提供更贴近业务的数据分析的服务。在数据服务至上抽象了多个数据应用。
+
+
+## 2.需求
+我们通过Presto
Connector和Spark组件，把底层的Hive、Kudu、Alluxio组件打通。大数据组件之间可以互相导入导出数据，可以根据数据分析的需求和场景任意利用合适的组件分析数据。但我们引入Clickhouse时，它是一个数据孤岛，数据的导入和导出比较困难。Hive和Clickhouse之间需要做很多工作才能实现导入导出。我们的第一个数据导入导出需求就是提升导入导出效率，把Clickhouse纳入大数据体系中。
+
+
+第二个需求是Presto跑SQL比较慢，图中是一个慢SQL的例子。图中的SQL
where条件设置了日期、时间范围和具体过滤条件，这类SQL使用由于Presto使用分区粒度下推，运行比较慢。即使用Hive的Bucket表和分桶等其他方式优化后也是几秒的返回时间、不能满足业务要求。这种情况下，我们需要利用Clickhouse做离线的OLAP计算加速。
+
+
+我们的实时数据是通过Kafka、Flink
SQL方式写入到Clickhouse中。但分析时只用实时数据是不够的，需要用Hive维度表和已经ETL计算号的T+1实时表一起在Clickhouse中做加速运输。这需要把Hive的数据导入到Clickhouse中，这就是我们的第三个需求。
+
+
+## 3.痛点
+首先，我们引入一项数据组件时要考虑其性能。Hive表粒度是五分钟，是否有组件可以支撑五分钟内完成一个短小ETL流程并把ETL结果导入到Clickhouse中？第二，我们需要保证数据质量，数据的准确性需要有保障。Hive和Clickhouse的数据条数需要保障一致性，如果数据质量出问题能否通过重跑等机制修复数据？第三，数据导入需要支持的数据类型是否完备？不同数据库之间的数据类型和一些机制不同，我们有HiperLogLog和BitMap这类在某一存储引擎中利用得比较多得数据类型，是否可以正确传输和识别，且可以较好地使用。
+
+# ClickHouse和Hive出仓入仓工具的选型
+基于数据业务上的痛点，我们对数据出仓入仓工具进行了对比和选择。我们主要在开源工具中进行选择，没有考虑商业出入仓工具，主要对比DataX、SeaTunnel和编写Spark程序并用jdbc插入ClickHouse这三个方案中取舍。
+SeaTunnel和Spark依赖唯品会自己的Yarn集群，可以直接实现分布式读取和写入。DataX是非分布式的，且Reader、Writer之间的启动过程耗时时间长，性能普通，SeaTunnel和Spark处理数据的性能可以达到DataX的数倍。
+十亿以上的数据可以平稳地在SeaTunnel和Spark中运行，DataX在数据量大以后性能压力大，处理十亿以上数据吃力。
+在读写插件扩展性方面，SeaTunnel支持了多种数据源，支持用户开发插件。SeaTunnel支持了数据导入Redis。
+稳定性上，SeaTunnel和DataX由于是自成体系的工具，稳定性会更好。Spark的稳定性方面需要关注代码质量。
+
+
+我们的曝光表数据量每天在几十亿级，我们有5min内完成数据处理的性能要求，我们我们存在数据导入导出到Redis的需求，我们需要导入导出工具可以接入到数据平台上进行任务调度。
出于数据量级、性能、可扩展性、平台兼容性几方面的考虑，我们选择了SeaTunnel作为我们的数仓导入导出工具。
+# Hive数据导入到ClickHouse
+下面将介绍我们对SeaTunnel的使用。
+图中是一张Hive表，它是我们三级的商品维度表，包含品类商品、维度品类和用户人群信息。表的主键是一个三级品类ct_third_id，下面的value是两个uid的位图，是用户id的bitmap类型，我们要把这个Hive表导入到Clickhouse。
+
+
+SeaTunnel安装简单，官网文档有介绍如何安装。下图中是SeaTunnel的配置，配置中env、source和sink是必不可少的。env部分，图中的例子是Spark配置，配置了包括并发度等，可以调整这些参数。source部分是数据来源，这里配置了Hive数据源，包括一条Hive
Select语句，Spark运行source配置中的SQL把数据读出，此处支持UDF进行简单ETL；sink部分配置了Clickhouse，可以看到output_type=rowbinary，rowbinary是唯品会自研加速方案；pre_sql和check_sql是自研的用于数据校验的功能，后面也会详细介绍；clickhouse.socket_timeout和bulk_size都是可以根据实际情况进行调整的。
+
+
+运行SeaTunnel，执行sh脚本文件、配置conf文件地址和yarn信息，后即可。
+
+运行过程中会产生Spark日志，运行成功和运行中错误都可以在日志中查看。
+
+
+为了更贴合业务，唯品会对SeaTunnel做了一些改进。我们的ETL任务都是需要重跑的，我们支持了pre_sql和check_sql实现数据的重跑和对数。主要流程是在数据准备好后，执行pre_sql进行预处理，在Clickhouse中执行删除旧分区数据、存放到某一目录下在失败时恢复该分区、rename这类操作。check_sql会检验，校验通过后整个流程结束；如果检验不通过，根据配置进行重跑，重跑不通过则报警到对应负责人。
+
+
+
+唯品会基于1.0版本SeaTunnel增加了RowBinary做加速，也让HuperLogLog和BinaryBitmap的二进制文件能更容易地从Hive导入到Clickhouse。我们在ClickHouse-jdbc、bulk_size、Hive-source几处进行了修改。使用CK-jdbc的extended
api，以rowbinary方式将数据写入CK，bulk_size引入了以rowbinary方式写入CK的控制逻辑，Hive-source
+RDD以HashPartitioner进行分区将数据打散，防止数据倾斜。
+
+我们还让SeaTunnel支持了多类型，为了圈人群的功能，需要在Clickhouse、Preso、Spark中实现对应的方法。我们在Clickhouse-jdbc中增加支持Batch特性的Callback、HttpEntity、RowBinaryStream，在Clickhouse-jdbc和Clickhouse-sink代码中增加了bitmap类型映射，在Presto和Spark中实现了Clickhouse的Hyperloglog和Bitmap的function的UDF。
+前面的配置中，Clickhouse-sink部分可以指定表名，这里有写入本地表和分布式表的差异。写入分布式表的性能比写入本地表差对Clickhouse集群的压力会更大，但在计算曝光表、流量表，ABTest等场景中需要两表Join，两张表量级均在几十亿。这时我们希望Join
key落在本机，Join成本更小。我们建表时在Clickhouse的分布式表分布规则中配置murmurHash64规则，然后在Seatunnel的sink里直接配置分布式表，把写入规则交给Clickhouse，利用了分布式表的特性进行写入。写入本地表对Clickhouse的压力会更小，写入的性能也会更好。我们在Seatunnel里，根据sink的本地表，去Clickhouse的System.cluster表里获取表的分布信息和机器分布host。然后根据均分规则写入这些host。把数据分布式写入的事情放到Seatunnel里来做。
+针对本地表和分布式表的写入，我们未来的改造方向是在Seatunnel实现一致性哈希，直接按照一定规则写如Clickhouse、不依赖Clickhouse自身做数据分发，改善Clickhouse高CPU负载问题。
+
+# ClickHouse数据导入到Hive
+我们有圈人群的需求，每天唯品会为供应商圈20万个人群，比如80后、高富帅、白富美的人群集合。这些在Clickhouse中的Bitmap人群信息需要导出到Hive表，在Hive中与其他ETL任务进行配合，最后推到PIKA交给外部媒体使用。我们使SeaTunnel将Clickhouse
Bitmap人群数据反推到Hive。
+
+
+图中是SeaTunnel配置，我们把source配置为Clickhouse、sink配置为Hive，数据校验也配置在Hive内。
+
+
+由于我们接入SeaTunnel较早，我们对一些模块间进行了加工，包括新增plugin-spark-sink-hive模块、plugin-spark-source-ClickHouse模块，重写Spark
Row相关方法，使其能封装经过Schem映射后的Clickhouse数据，重新构造StructField并生成最终需要落地Hive的DataFrame。最新版本已经有了很多source和sink组件，在SeaTunnel使用上更方便。现在也可以在SeaTunnel中直接集成Flink
connector。
+
+# SeaTunnel与唯品会数据平台的集成
+各个公司都有自己的调度系统，例如白鲸、宙斯。唯品会的调度工具是数坊，调度工具中集成了数据传输工具。下面是调度系统架构图，其中包含各类数据的出入仓。
+
+
+SeaTunnel任务类型集成到平台中，图中是数坊的定时任务截图，可以看到选中的部分，是一个配置好的SeaTunnel任务，负责人、最近一次耗时，前后依赖任务的血缘信息，消耗的资源信息。下面展示了历史运行实例信息。
+
+
+我们把SeaTunnel集成到了调度系统中，数坊调度Master会根据任务类型把任务分配到对应的Agent上，根据Agent负载情况分配到合适的机器上运行，管控器把前台的任务调度配置和信息拉取到后生成SeaTunnel
cluster，在类似于k8s
pod、cgroup隔离的虚拟环境内进行执行。运行结果会由调度平台的数据质量监控判断任务是否完成、是否运行成功，失败时进行重跑和告警。
+
+
+SeaTunnel本身是一个工具化的组件，是为了进行数据血缘，数据质量，历史记录，高警监控，还包括资源分配这些信息的管控。我们把SeaTunnel集成到平台中，可以利用平台优势利用好SeaTunnel。
+圈存人群中利用了SeaTunnel进行处理。我们通过打点数据，把圈存人群按照路径和使用情况分为不同的人，或称千人千面，把用户打上标签，圈出的某一类人群推送给用户、分析师和供应商。
+
+
+流量进入Kafka，通过Flink入仓，再通过ETL形成用户标签表，用户标签表生成后，我们通过Presto实现了的BitMap方法，把数据打成Hive中的宽表。用户通过在人群系统页面中框选词条创建任务，提交腾群，生成SQL查询Clickhouse
BitMap。Clickhouse的BitMap查询速度非常快，由天生优势，我们需要把Hive的BitMap表通过SeaTunnel导入到Clickhouse中。圈完人群后我们需要把表落地，形成Clickhouse的一个分区或一条记录，再把生成的结果BitMap表通过SeaTunnel存储到Hive中去。最后同步工具会将Hive的BitMap人群结果同步给外部媒体仓库Pika。每天圈20w个人群左右。
+整个过程中SeaTunnel负责把数据从Hive导出到Clickhouse，Clickhouse的ETL流程完成后SeaTunnel把数据从Clickhouse导出到Hive。
+为了完成这样的需求，我们在Presto和Spark端现ClickHouse的Hyperloglog和BitMap的function的UDF；我们还开发Seatunnel接口，使得用户在ClickHouse里使用Bitmap方法圈出来的人群，可以直接通过Seatunnel写入Hive表，无需中间落地步骤。用户也可以在Hive里通过spark圈人群或者反解人群bitmap，调用SeaTunnel接口，使数据直接传输到ClickHouse的结果表，而无需中间落地。
+# 后续工作
+后续我们会进一步改善Clickhouse写入数据时CPU负载高的问题，下一步会在SeaTunnel中实现Clickhouse数据源和读取端的CK-local模式，读写分离，减轻Clickhouse压力。未来我们也会增加更多sink支持，如数据推送到Pika和相应的数据检查。
diff --git a/i18n/zh-CN/docusaurus-theme-classic/navbar.json
b/i18n/zh-CN/docusaurus-theme-classic/navbar.json
index 070509220668..bf10cf7e24dc 100644
--- a/i18n/zh-CN/docusaurus-theme-classic/navbar.json
+++ b/i18n/zh-CN/docusaurus-theme-classic/navbar.json
@@ -27,10 +27,6 @@
"message": "博客",
"description": "Navbar item with label BLOG"
},
- "item.label.UserCases": {
- "message": "用户案例",
- "description": "Navbar item with label UserCases"
- },
"item.label.Team": {
"message": "团队",
"description": "Navbar item with label Team"
@@ -75,4 +71,4 @@
"message": "致谢",
"description": "Navbar item with label Thanks"
}
-}
\ No newline at end of file
+}
