This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git
commit 6df11c8cdf1a00da253b694edca6bc2396fadaf7 Author: jiafeng.zhang <[email protected]> AuthorDate: Mon Nov 28 15:12:57 2022 +0800 NIO --- blog/NIO.md | 262 +++++++++++++++++++++++ i18n/zh-CN/docusaurus-plugin-content-blog/NIO.md | 223 +++++++++++++++++++ static/images/NIO_kv.png | Bin 0 -> 446037 bytes static/images/cdp.png | Bin 0 -> 58066 bytes static/images/olap.png | Bin 0 -> 81456 bytes 5 files changed, 485 insertions(+) diff --git a/blog/NIO.md b/blog/NIO.md new file mode 100644 index 00000000000..3591c6ee684 --- /dev/null +++ b/blog/NIO.md @@ -0,0 +1,262 @@ +--- +{ + 'title': 'The Application of Apache Doris in NIO', + 'summary': "NIO Inc. (NYSE: NIO)is a leading company in the premium smart electric vehicle market. Founded in November 2014, NIO designs, develops, jointly manufactures and sells premium smart electric vehicles, driving innovations in autonomous driving, digital technologies, electric powertrains and batteries. +Recently, NIO planned to enter the U.S. market alongside other western markets by the end of 2025. The company has already established a U.S. headquarters in San Jose, California, where they started hiring people..", + 'date': '2022-11-28', + 'author': 'Huaidong Tang', + 'tags': ['Best Practice'], +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +# The Application of Apache Doris in NIO + + + +>Guide: The topic of this sharing is the application of Apache Doris in NIO, which mainly includes the following topics: +>1. Introduction about NIO +>2. The Development of OLAP in NIO +>3. Apache Doris-the Unified OLAP Data warehouse +>4. Best Practice of Apache Doris on CDP Architecture +>5. Summery and Benefits + +Author：Huaidong Tang, Data Team Leader, NIO INC + +## About NIO + +NIO Inc. (NYSE: NIO)is a leading company in the premium smart electric vehicle market. Founded in November 2014, NIO designs, develops, jointly manufactures and sells premium smart electric vehicles, driving innovations in autonomous driving, digital technologies, electric powertrains and batteries. + +Recently, NIO planned to enter the U.S. market alongside other western markets by the end of 2025. The company has already established a U.S. headquarters in San Jose, California, where they started hiring people. + +## The Architecture Evolution of OLAP in NIO + +The architectural evolution of OLAP in NIO took several steps for years. + +### 1. Introduced Apache Druid + +At that time, there were not so many OLAP storage and query engines to choose from. The more common ones were Apache Druid and Apache Kylin. There are 2 reasons why we didn't choose Kylin. + +- The most suitable and optimal storage at the bottom of Kylin is HBase and adding it would increase the cost of operation and maintenance. + +- Kylin's precalculation involves various dimensions and indicators. Too many dimensions and indicators would cause great pressure on storage. + +We prefer Druid because we used to be users and are familiar with it. Apache Druid has obvious advantages. It supports real-time and offline data import, columnar storage, high concurrency, and high query efficiency. But it has downsides as well: + +- Standard protocols such as JDBC are not used + +- The capability of JOIN is weak + +- Significant performance downhill when performing dedeplication + +- High in operation and maintenance costs, different components have separate installation methods and different dependencies; Data import needs extra integration with Hadoop and the dependencies of JAR packages + +### 2. Introduced TiDB + +**TiDB is a mature datawarehouse focused on OLTP+OLAP, which also has distinctive advantages and disadvantages:** + +Advantage: + +- OLTP database, can be updated friendly + +- Supports detailed and aggregated query, which can handle dashboard statistical reports or query of detailed data at the same time + +- Supports standard SQL, which has low cost of use + +- Low operation and maintenance cost + +Disadvantages: + +- It is not an independent OLAP. TiFlash relies on OLTP and will increase storage. Its OLAP ability is insufficient + +- The overall performance should be measured separately by each scene + +### 3. Introduced Apache Doris + +Since 2021, we have officially introduced Apache Doris. In the process of selection, we are most concerned about various factors such as product performance, SQL protocol, system compatibility, learning and operation and maintenance costs. After deep research and detailed comparison of the following systems, we came to the following conclusions: + +**Apache Doris, whose advantages fully meet our demands:** + +- Supports high concurrent query (what we concerned most) + +- Supports both real-time and offline data + +- Supports detailed and aggregated query + +- UNIQ model can be updated + +- The ability of Materialized View can greatly speed up query efficiency + +- Fully compatible with the MySQL protocol and the cost of development is relatively low + +- The performance fully meets our requirements + +- Lower operation and maintenance costs + +**Moreover, there is another competitor, Clickhouse. Its stand-alone performance is extremely strong, but its disadvantages are hard to accept:** + +- In some cases, its multi-table JOIN is weak + +- Relatively low in concurrency + +- High operation and maintenance costs + +With multiple good performances, Apache Doris outstands Druid and TiDB. Meanwhile Clickhouse did not fit well in our business, which lead us to Apache Doris. + +## Apache Doris-the Unified OLAP Datawarehouse + + +This diagram basically describes our OLAP Architecuture, including data source, data import, data processing, data warehouse, data service and application. + +### 1. Data Source + +In NIO, the data source not only refers to database, but also event tracking data, device data, vehicle data, etc. The data will be ingested into the big data platform. +### 2. Data Import + +For business data, you can trigger CDC and convert it into a data stream, store it in Kafka, and then perform stream processing. Some data that can only be passed in batches will directly enter our distributed storage. + +### 3. Data Processing + +We took the Lambda architecture rather than stream-batch integration. + +Our own business determines that our Lambda architecture should be divided into two paths: offline and real-time: + +- Some data is streamed. + +- Some data can be stored in the data stream, and some historical data will not be stored in Kafka. + +- Some data requires high precision in some circumstances. In order to ensure the accuracy of the data, an offline pipeline will recalculate and refresh the entire data. + +### 4. Data Warehouse + +From data processing to the data warehouse, we did not adopt Flink or Spark Doris Connector. We use Routine Load to connect Apache Doris and Flink, and Broker Load to connect Doris and Spark. The data generated in batches by Spark will be backed up to Hive for further use in other scenarios. In this way, each calculation is used for multiple scenarios at the same time, which greatly improves the efficiency. It also works for Flink. + +### 5. Data Service + +What behind Doris is One Service. By registering the data source or flexible configuration, the API with flow and authority control is automatically generated, which greatly improves flexibility. And with the k8s serverless solution, the entire service is much more flexible. + +### 6. Application + +In the application layer, we mainly deploy some reporting applications and other services. + +We mainly have two types of scenarios: + +- **User-oriented** , which is similar to the Internet, contains a data dashboard and data indicators. + +- **Car-oriented** , car data enters Doris in this way. After certain aggregation, the volume of Doris data is about billions. But the overall performance can still meet our requirements. + +## Best Practice of Apache Doris on CDP Architecture + +### 1. CDP Architecture + + + +Next, let me introduce Doris' practice on the operating platform. This is what happens in our real business. Nowadays, Internet companies will make their own CDP, which includes several modules: + +- **Tags** , which is the most basic part. + +- **Target** , based on tags, select people according to some certain logic. + +- **Insight** , aiming at a group of people, clarify the distribution and characteristics of the group. + +- **Touch** , use methods such as text messages, phone calls, voices, APP notifications, IM, etc. to reach users, and cooperate with flow control. + +- **Effect analysis,** to improve the integrity of the operation platform, with action, effect and feedback. + +Doris plays the most important role here, including: tags storage, groups storage, and effect analysis. + +Tags are divided into basic tags and basic data of user behavior. We can flexibly customize other tags based on those facts. From the perspective of time effectiveness, tags are also divided into real-time tags and offline tags. + +### 2. Considerations for CDP Storage Selection + +We took five dimensions into account when we select CDP storage. + +**(1) Unification of Offline and Real-time** + +As mentioned earlier, there are offline tags and real-time tags. Currently we are close to quasi-real-time. For some data, quasi-real-time is good enough to meet our needs. A large number of tags are still offline tags. The methods used are Doris's Routine Load and Broker Load. + +| **Scenes** | **Requirements** | **Apache Doris's Function** | +| --- | --- | --- | +| Real-time tags | Real-time data updates | Routine Load | +| Offline tags | Highly efficient batch import | Broker Load | +| Unification of offline and real-time | Unification of offline and real-time data storage | Routine Load and Broker Load update different columns of the same table | + +In addition, on the same table, the update frequency of different columns is also different. For example, we need to update the user's identity in real time because the user's identity changes all the time. T+1's update does not meet our needs. Some tags are offline, such as the user's gender, age and other basic tags, T+1 update is sufficient to meet our standards. The maintenance cost caused by putting the tags of basic users on the same table is very low. When customizing tags later, [...] + +**(2) Efficient Targets** + +When users tags are done, is time to target right group of people. The target is to filter out all the people who meet the conditions according to different combinations of tags. At this time, there will be queries with different combinations of tag conditions. There was an obvious improvement when Apache Doris upgraded to vectorization. + +| **Scenes** | **Requirements** | **Apache Doris's Function** | +| --- | --- | --- | +| Complex Condition Targets | Highly efficient combination of tags | Optimization of SIMD | + +**(3) Efficient Polymerization** + +The user insights and effect analysis statistics mentioned above require statistical analysis of the data, which is not a simple thing of obtaining tags by user ID. The amount of data read and query efficiency have a great impact on the distribution of our tags, the distribution of groups, and the statistics of effect analysis. Apache Doris helps a lot: + +- Data Partition. We shard the data by time order and the analysis and statistics will greatly reduce the amount of data, which can greatly speed up the efficiency of query and analysis. + +- Node aggregation. Then we collect them for unified aggregation. + +- Vectorization. The vectorization execution engine has significant performance improvement. + +| **Scenes** | **Requirements** | **Apache Doris's Function** | +| --- | --- | --- | +| Distribution of Tags Values | The distribution values of all tags need to be updated every day. Fast and efficient statistics are required | Data partition lessens data transfer and calculation | +| Distribution of Groups | Same as Above | Unified storage and calculation, each node aggregates first | +| Statistics for Performance Analysis | Same as Above | Speed up SIMD + + +**(4) Multi-table Association** + +Our CDP might be different from common CDP scenarios in the industry, because common CDP tags in some scenarios are estimated in advance and no custom tags, which leaves the flexibility to users who use CDP to customize tags themselves. The underlying data is scattered in different database tables. If you want to create a custom tag, you must associate the tables. + +A very important reason we chose Doris is the ability to associate multiple tables. Through performance tests, Apache Doris is able to meet our requirements. And Doris provides users with powerful capabilities because tags are dynamic. + +| **Scenes** | **Requirements** | **Apache Doris's Function** | +| --- | --- | --- | +| Distributed Characteristics of the Population | The distribution of statistical groups under a certain characteristic | Table Association | +| Single Tag | Display tags | + +**(5) Query Federation** + +Whether the user is successfully reached or not will be recorded in TiDB. Notifications during operations may only affect user experience. If a transaction is involved, such as gift cards or coupons, the task execution must be done without repetition. TiDB is more suitable for this OLTP scenario. + +But for effect analysis, it is necessary to understand the extent to which the operation plan is implemented, whether the goal is achieved and its distribution. It is necessary to combine task execution and group selection for analysis, which requires the query association between Doris and TiDB. + +The size of the tag is probably small, so we would like to save it into Elasticsearch. However, it proves us wrong later. + +| **Scenes** | **Requirements** | **Apache Doris's Function** | +| --- | --- | --- | +| Effect Analysis Associated with Execution Details | Doris query associated with TiDB | Query Association with other databases | +| Group Tags Associated with Behavior Aggregation | Doris query associated with Elasticsearch | + +## Summery and Benefits + +1. **bitmap**. Our volume are not big enough to test its full efficiency. If the volume reaches a certain level, using bitmap might have a good performance improvement. For example, when calculating UV , bitmap aggregation can be considered if the full set of Ids is greater than 50 million. + +2. **The performance is good** when Elasticsearch single-table query is associated with Doris. + +3. **Better to update columns in batches**. In order to reduce the number of tables and improve the performance of the JOIN table, the table designed should be as streamlined as possible and aggregated as much as possible. However, fields of the same type may have different update frequencies. Some fields need to be updated at daily level, while others may need to be updated at hourly level. Updating a column alone is an important requirement. The solution from Apache Doris is to use REP [...] + +4. **Online Services**. Apache Doris serves online and offline scenarios at the same time, which requires high resource isolation. diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/NIO.md b/i18n/zh-CN/docusaurus-plugin-content-blog/NIO.md new file mode 100644 index 00000000000..86072037c01 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-blog/NIO.md @@ -0,0 +1,223 @@ +--- +{ + 'title': 'Apache Doris 在蔚来汽车的应用', + 'summary': '蔚来（纽约证券交易所代码：NIO）是设计高端智能电动汽车市场的领先公司. NIO 成立于 2014 年 11 月，设计、开发、联合制造和销售高端智能电动汽车，并不断推动自动驾驶、数字技术、电动动力总成和电池领域的创新,Doris作为统一OLAP数仓,Doris在运营平台上的实践', + 'date': '2022-11-28', + 'author': '唐怀东', + 'tags': ['最佳实践'], +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +# Apache Doris 在蔚来汽车的应用 + + + +>导读：本次分享的题目是Apache Doris在蔚来汽车的应用，主要包括以下几大部分： +>1. 蔚来 +>2. OLAP在蔚来的发展 +>3. Doris作为统一OLAP数仓 +>4. Doris在运营平台上的实践 +>5. 经验总结 + +作者：唐怀东，蔚来汽车 数据团队负责人 + +## 蔚来 + +蔚来（纽约证券交易所代码：NIO）是设计高端智能电动汽车市场的领先公司。 NIO 成立于 2014 年 11 月，设计、开发、联合制造和销售高端智能电动汽车，并不断推动自动驾驶、数字技术、电动动力总成和电池领域的创新。 + +## OLAP在蔚来的发展 + +首先，让我们来一起回顾OLAP在蔚来汽车的发展。 + +### 1. 2017年引入Apache Druid + +在当时可选择的OLAP存储和查询引擎并不多，比较常见的有Apache Druid、Apache Kylin。我们优先引入Druid的原因是以前有使用经验，而Kylin预计算虽然具有极高的查询效率优势，但是： + +- Kylin底层最合适和最优的存储是HBase，之前公司并未引入，会额外增加运维的工作。 + +- Kylin对各种维度和指标进行预计算，如果维度和维度取值非常多，会有维度爆炸的问题，对存储造成非常大的压力。 + +Druid的优势很明显，支持实时和离线数据接入，列式存储，高并发，查询效率非常高。其缺点也比较明显： + +- 未使用标准协议例如JDBC，使用门槛高 +- Join的支持较弱 +- 精确去重的效率低，性能会随之下降。整体性能要分场景去考虑，这也是我们后期去选型其他OLAP的原因 +- 运维成本高，不同的组件有不同的安装方式和不同的依赖；数据导入还要考虑和Hadoop集成以及JAR包的依赖 + +### 2. 2019年引入TiDB + +**TiDB是一个OLTP+OLAP的成熟引擎，同样是优点、缺点分明：** + +优势： + +- OLTP数据库，更新友好。 +- 支持明细和聚合，有指标计算和数据看板展示，还支持明细数据查询 +- 支持标准SQL，使用成本低 +- 运维成本低 + +劣势： +- 它不是一个独立的OLAP。TiFlash依赖于OLTP，会增加存储。其OLAP能力稍显不足 +- 整体性能要分场景去衡量 + +### 3. 2021年引入Doris + +自2021年起，我们正式引入了Apache Doris。在系统选型过程中，产品的性能、SQL语法、系统兼容性、学习以及运维成本等多方面因素是我们最为关心的部分。经过深入调研、层层对比以下几个系统后，我们得出了如下结论： + +**我们重点关注的Doris，其优点完全满足我们的诉求：** + +- 支持高并发查询（我们最关心的一点） +- 同时支持实时和离线数据 +- 支持明细和聚合 +- Uniq模型支持更新 +- 物化视图的能力能极大的加速查询效率 +- 兼容MySQL协议，所以开发和使用成本比较低 +- 性能完全满足我们的要求 +- 运维成本比较低 + +**Clickhouse，我们之前也调研过，也尝试想去使用它。其单机性能极强，但是缺点明显:** + +- 我们明确需要的场景下，它的多表join支持的稍微差一些 +- 并发度比较低 +- 运维成本极高 + +凭借多种性能优势，Apache Doris比较理想地替代了Druid和TiDB。而Clickhouse在我们的业务上并不能很好的适配，让我们最终走向了Apache Doris。 + +## Doris作为统一OLAP数仓 + + + +这张图基本上就是从数据源到数据接入、数据计算、数据仓库、数据服务以及应用。 + +### 1. 数据源 + +蔚来的场景下，数据源不仅仅指业务系统的数据，还有埋点数据、设备数据、车辆数据等等。数据会通过一种接入方式接入到大数据平台。 + +### 2. 数据接入 + +对于一些业务系统的数据，可以开启CDC捕捉变化的数据，然后转换成一个数据流存储到Kafka，接续再进行流式的计算。某些只能通过批量的方式的数据会直接进入到我们的分布式存储。 + +### 3. 数据计算 + +我们没有采用流批一体，采用的是Lambda架构。 +我们本身的业务决定了我们的Lambda架构是离线和实时分成了两条路径： +- 部分数据是流式的。 +- 部分数据能够存储到数据流里，一些历史数据不会存储到Kafka。 +- 有些场景数据要求高精准度。为了保证数据的准确性，一个离线的pipeline将会把整个数据重新计算和刷新。 + +### 4. 数据仓库 + +数据计算到数仓，这两条线路我们没有采用Flink或Spark Doris Connector。我们用Routine Load来连接Apache Doris和Flink，用Broker Load连接Doris和Spark。 由Spark批量生成的数据，会备份到Hive供其他场景使用。这样每计算一次，就同时供多个场景去使用，大大提升了效率。Flink的情况也诸如此类。 + +### 5. 数据服务 + +Doris后面是One Service。通过注册数据源或灵活配置的方式，自动生成API，对API进行流量的控制和权限的控制，灵活性大大提高。并借助于k8s serverless方案，整个服务非常灵活和丰富。 + +### 6. 数据应用 + +应用层中我们主要是部署一些报表应用和其他的一些服务。 + +我们主要有两类使用场景： +- 面向用户，类似于互联网，我们有很多用户的场景，包括看板和指标 +- 面向车，车的数据通过这种方式进入到Doris，通过一定的聚合之后，Doris数据体量在几十亿级别。但总体性能仍然可以满足我们的要求。 + +## Doris在运营平台上的实践 + +### 1. CDP Architecture + + + +接下来我来介绍Doris在运营平台上的实践。这是我们的真实使用场景。如今互联网公司普遍会做自己的CDP，它一般包括几个模块： +- 标签，是最基础的部分。 +- 圈人，基于标签，按照一定逻辑将人圈选出来。 +- 洞察，针对圈定的人群，了解人群分布、特点。 +- 触达，利用例如短信、电话、声音、APP通知、IM等方式触达到用户，并配合流量控制。 +- 效果分析，提升运营平台的完整性，有动作、有效果、有反馈。 + +Doris在这里面起到了最重要的作用，包括：标签存储、人群存储、效果分析。 +标签分为基础标签和用户行为的基础数据，在此基础之上，我们可以灵活自定义其他标签。从实效性来看，标签还分为实时的标签和离线的标签。 + +### 2. CDP存储选型的考量点 +我们从5个维度去考量CDP存储的选型。 + +**(1) 离线和实时统一 +如前所述标签有离线标签，有实时标签。目前我们是准实时的场景。对于有些数据，准实时已足够满足我们的需求，大量的标签还是离线的标签，采用的方式就是Doris的Routine Load和Broker Load。 + +| **场景** | **需求** | **Apache Doris功能点** | +| --- | --- | --- | +| 实时标签 | 数据实时更新 | Routine Load | +| 离线标签 | 高效大批量导入 | Broker Load | +| 流批统一 | 实时历险数据存储统一 | Routine Load 和 Broker Load 更新同一张表的不同列 | + +另外同一张表上，不同列更新的频率也是不一样的。例如用户的基础标签，我们对用户的身份需要实时的更新，因为用户的身份是时刻变化的。T+1的更新不能满足我们的需求。有些标签离线，例如用户的性别、年龄等基础标签，T+1更新足以满足我们的标准。基础用户的原子标签放在一张表中带来的维护成本很低。当后期自定义标签时，表的数量会大大减少，这样对于整体性能的提升有极大好处。 + +**(2) 高效圈选** + +用户运营有了标签，第二步就是圈人，圈选就是根据标签的不同组合，把符合标签条件的所有人筛选出来，这时会有不同标签条件组合的查询、这个查询在Doris引入向量化之后有比较明显的提升。 + +| **场景** | **需求** | **Apache Doris功能点** | +| --- | --- | --- | +| 复杂条件圈选 | 高效的支持多条件圈选 | SIMD的优化 | + +**(3) 高效聚合** + +前面提到的用户洞察或群体洞察以及效果分析统计，需要对数据做统计分析，并不是单一的按用户ID获取标签的这种简单场景。其读取的数据量和查询效率，对我们这个标签的分布、群体的分布、效果分析的统计都有很大的影响。在这里，体现到的Doris的功能特点是： +- 第一是数据分片，我们按时间把数据分片，分析统计就会极大的减少数据量，可以极大的加速查询和分析的效率。 +- 第二是节点聚合，然后再收集做统一的聚合。 +- 第三是向量化加速，向量化引擎对性能提升非常显著。 + +| **场景** | **需求** | **Apache Doris功能点** | +| --- | --- | --- | +| 标签值的分布 | 每天都需要更新所有标签，需要快速高效统计 | 数据分片，减少数据传输和计算 | +| 群体的分布 | 同上 | 存算统一，每个节点先聚合 | +| 效果分析的统计值 | 同上 | SIMD提速 | + +**(4) 多表关联** + +我们的CDP可能和业内常见的CDP场景不太一样，因为有些场景的CDP标签是提前预估完成的，不存在自定义标签。只做原子标签，或者说用户基础行为数据的统计，这样可以把灵活性留给使用CDP的用户，根据自己的业务场景去自定义标签。底层的数据是分散在不同的数据库表里，如果做自定义的标签的建设，势必需要做表的关联。 +我们选择Doris一个非常重要的原因，就是多表关联的能力。通过性能测试，Doris目前能够满足我们的要求。而且Doris为用户提供了非常强大的能力。因为标签是动态的。 + +| **场景** | **需求** | **Apache Doris功能点** | +| --- | --- | --- | +| 群体的特征分布 | 统计群体在某个特征下的分布 | 多表关联 | +| Single Tag | Display tags | | + +**(5) 联邦查询** + +用户触达成功与否我们会记录到TiDB。用户运营中的通知，可能只影响用户体验，如果涉及到钱例如发放积分或优惠券，任务执行就要做到不重不漏，这种OLTP场景用TiDB比较合适。 +做效果分析，需要了解运营计划执行到什么程度，是否达成目标，其分布情况等等。需要把任务执行情况和人群圈选相结合才能进行分析，就会用到Doris和TiDB的关联，外表关联进行查询。 +我们设想标签体量比较小，保存到es可能比较合适，然而ES不能满足我们的需求，后面会解释其原因。 + +| **场景** | **需求** | **Apache Doris功能点** | +| --- | --- | --- | +| 效果分析关联任务执行明细 | Doris数据关联TiDB数据 | 关联外表进行查询 | +| 人群标签关联行为聚合数据 | Doris数据关联Elasticsearch数据 | + +## 经验和总结 + +1. **bitmap**. 我们的体量无法充分发挥其效率。如果体量达到一定程度，用bitmap会有很好的性能提升。例如计算UV场景，Id全集大于5000万，可以考虑bitmap聚合。 + +2. **ES外表。单表查询下效率比较理想。** + +3. **分批更新列**. 为了减少表的数量和提升join表的性能，设计表尽量精简尽量聚合，相同类型的事实都放在一起。但相同类型的字段可能更新频率不同，有些字段需要天级更新，有些字段可能需要小时级更新，单独更新某一列就是一个明显的诉求。Doris聚合模型单独更新某些列的解决方案是使用REPLACE_IF_NOT_NULL。注意:用null替换原来的非null值是做不到的,可以把所有的null替换成有意义的默认值，例如unknown。 + +4. **在线服务**. Doris同一份数据同时服务在线离线场景，对资源隔离的要求比较高，目前还存在进一步优化的空间。 diff --git a/static/images/NIO_kv.png b/static/images/NIO_kv.png new file mode 100644 index 00000000000..f1c7aff1b09 Binary files /dev/null and b/static/images/NIO_kv.png differ diff --git a/static/images/cdp.png b/static/images/cdp.png new file mode 100644 index 00000000000..767356437b8 Binary files /dev/null and b/static/images/cdp.png differ diff --git a/static/images/olap.png b/static/images/olap.png new file mode 100644 index 00000000000..08b95f57e6a Binary files /dev/null and b/static/images/olap.png differ --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
