This is an automated email from the ASF dual-hosted git repository. hui pushed a commit to branch lmh/UpdateQueryConfig in repository https://gitbox.apache.org/repos/asf/iotdb.git
commit 8d9222265e82d6e8a738d126e5e43882ad83c640 Author: liuminghui233 <[email protected]> AuthorDate: Fri Nov 25 21:38:35 2022 +0800 update --- docs/UserGuide/Reference/Common-Config-Manual.md | 119 ++++++++++---- .../zh/UserGuide/Reference/Common-Config-Manual.md | 181 ++++++--------------- .../resources/conf/iotdb-common.properties | 72 +++----- 3 files changed, 156 insertions(+), 216 deletions(-) diff --git a/docs/UserGuide/Reference/Common-Config-Manual.md b/docs/UserGuide/Reference/Common-Config-Manual.md index 6b67107746..dedca077cb 100644 --- a/docs/UserGuide/Reference/Common-Config-Manual.md +++ b/docs/UserGuide/Reference/Common-Config-Manual.md @@ -251,6 +251,50 @@ IoTDB common files for ConfigNode and DataNode are under `conf`. ### Query Configurations +* read\_consistency\_level + +| Name | mpp\_data\_exchange\_core\_pool\_size | +|:-----------:|:---------------------------------------------| +| Description | The read consistency level, </br>1. strong(Default, read from the leader replica) </br>2. weak(Read from a random replica) | +| Type | string | +| Default | strong | +| Effective | After restarting system | + +* meta\_data\_cache\_enable + +|Name| meta\_data\_cache\_enable | +|:---:|:---| +|Description| Whether to cache meta data(BloomFilter, ChunkMetadata and TimeSeriesMetadata) or not.| +|Type|Boolean| +|Default| true | +|Effective| After restarting system| + +* chunk\_timeseriesmeta\_free\_memory\_proportion + +|Name| chunk\_timeseriesmeta\_free\_memory\_proportion | +|:---:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +|Description| Read memory Allocation Ratio: BloomFilterCache : ChunkCache : TimeSeriesMetadataCache : Coordinator : Operators : DataExchange : timeIndex in TsFileResourceList : others. | +|Default| 1 : 100 : 200 : 300 : 400 | +|Effective| After restarting system | + +* enable\_last\_cache + +|Name| enable\_last\_cache | +|:---:|:---| +|Description| Whether to enable LAST cache. | +|Type| Boolean | +|Default| true | +|Effective|After restarting system| + +* max\_deduplicated\_path\_num + +|Name| max\_deduplicated\_path\_num | +|:---:|:---| +|Description| allowed max numbers of deduplicated path in one query. | +|Type| Int32 | +|Default| 1000 | +|Effective|After restarting system| + * mpp\_data\_exchange\_core\_pool\_size | Name | mpp\_data\_exchange\_core\_pool\_size | @@ -305,23 +349,50 @@ IoTDB common files for ConfigNode and DataNode are under `conf`. |Default| 1000 | |Effective|After restarting system| -* default\_fill\_interval +* slow\_query\_threshold + +|Name| slow\_query\_threshold | +|:---:|:---| +|Description| Time cost(ms) threshold for slow query. | +|Type| Int32 | +|Default| 5000 | +|Effective|Trigger| + +* query\_timeout\_threshold + +|Name| query\_timeout\_threshold | +|:---:|:---| +|Description| The max executing time of query. unit: ms | +|Type| Int32 | +|Default| 60000 | +|Effective| After restarting system| + +* max\_allowed\_concurrent\_queries + +|Name| max\_allowed\_concurrent\_queries | +|:---:|:---| +|Description| The maximum allowed concurrently executing queries. | +|Type| Int32 | +|Default| 1000 | +|Effective|After restarting system| + +* query\_thread\_count -| Name | default\_fill\_interval | -| :---------: | :---------------------------------------------- | -| Description | Default interval of `group by fill` query in ms | -| Type | Int32 | -| Default | -1 | -| Effective | After restarting system | +|Name| query\_thread\_count | +|:---:|:---------------------------------------------------------------------------------------------------------------------| +|Description| How many threads can concurrently execute query statement. When <= 0, use CPU core number. | +|Type| Int32 | +|Default | CPU core number | +|Effective| After restarting system | -* group_by_fill_cache_size_in_mb +* batch\_size -| Name | group_by_fill_cache_size_in_mb | -| :---------: | :---------------------------------- | -| Description | Cache size of `group by fill` query | -| Type | Float | -| Default | 1.0 | -| Effective | After restarting system | +|Name| batch\_size | +|:---:|:---| +|Description| The amount of data iterate each time in server (the number of data strips, that is, the number of different timestamps.) | +|Type| Int32 | +|Default| 100000 | +|Effective|After restarting system| ### Storage Engine Configuration @@ -424,24 +495,6 @@ IoTDB common files for ConfigNode and DataNode are under `conf`. |Default| 0 | |Effective|After restarting system| -* query\_thread\_count - -|Name| query\_thread\_count | -|:---:|:---------------------------------------------------------------------------------------------------------------------| -|Description| The thread number which can concurrently execute query statement. When <= 0, use CPU core number. The default is 16. | -|Type| Int32 | -|Default| 16 | -|Effective| After restarting system | - -* sub\_rawQuery\_thread\_count - -|Name| sub\_rawQuery\_thread\_count | -|:---:|:-------------------------------------------------------------------------------------------------------------------------| -|Description| The thread number which can concurrently read data for raw data query. When <= 0, use CPU core number. The default is 8. | -|Type| Int32 | -|Default| 8 | -|Effective| After restarting system | - * enable\_partial\_insert |Name| enable\_partial\_insert | @@ -745,8 +798,6 @@ IoTDB common files for ConfigNode and DataNode are under `conf`. |Default| 1024 | |Effective|Trigger| -### Watermark Configuration - ### Authorization Configuration * authorizer\_provider\_class diff --git a/docs/zh/UserGuide/Reference/Common-Config-Manual.md b/docs/zh/UserGuide/Reference/Common-Config-Manual.md index f7f66d8d84..03c486233f 100644 --- a/docs/zh/UserGuide/Reference/Common-Config-Manual.md +++ b/docs/zh/UserGuide/Reference/Common-Config-Manual.md @@ -377,11 +377,20 @@ IoTDB ConfigNode 和 DataNode 的通用配置参数位于 `conf` 目录下。 ### 查询配置 +* read\_consistency\_level + +|名字| read\_consistency\_level | +|:---:|:---| +|描述| 查询一致性等级，取值 “strong” 时从 Leader 副本查询，取值 “weak” 时随机查询一个副本。| +|类型| String | +|默认值| strong | +|改后生效方式| 重启服务生效 | + * meta\_data\_cache\_enable |名字| meta\_data\_cache\_enable | |:---:|:---| -|描述| 是否缓存元数据Chunk Metadata 和 TimeSeries Metadata）。| +|描述| 是否缓存元数据（包括 BloomFilter、Chunk Metadata 和 TimeSeries Metadata。）| |类型|Boolean| |默认值| true | |改后生效方式| 重启服务生效| @@ -404,13 +413,13 @@ IoTDB ConfigNode 和 DataNode 的通用配置参数位于 `conf` 目录下。 |默认值| true | |改后生效方式|重启服务生效| -* metadata\_node\_cache\_size +* max\_deduplicated\_path\_num -|名字| metadata\_node\_cache\_size | +|名字| max\_deduplicated\_path\_num | |:---:|:---| -|描述| SchemaRegion的缓存大小。所有路径检查和将具有相应路径的SchemaRegion中的TSDataType的缓存，都将被用作提高写入速度。| -|类型|Int32| -|默认值| 300000 | +|描述| 单次查询允许的最大路径数。 | +|类型| Int32 | +|默认值| 1000 | |改后生效方式|重启服务生效| * mpp\_data\_exchange\_core\_pool\_size @@ -418,7 +427,7 @@ IoTDB ConfigNode 和 DataNode 的通用配置参数位于 `conf` 目录下。 | 名字 | mpp\_data\_exchange\_core\_pool\_size | |:------:|:--------------------------------------| | 描述 | MPP 数据交换线程池核心线程数 | -| 类型 | int | +| 类型 | Int32 | | 默认值 | 10 | | 改后生效方式 | 重启服务生效 | @@ -427,7 +436,7 @@ IoTDB ConfigNode 和 DataNode 的通用配置参数位于 `conf` 目录下。 | 名字 | mpp\_data\_exchange\_max\_pool\_size | |:------:|:--------------------------------------| | 描述 | MPP 数据交换线程池最大线程数 | -| 类型 | int | +| 类型 | Int32 | | 默认值 | 10 | | 改后生效方式 | 重启服务生效 | @@ -436,34 +445,16 @@ IoTDB ConfigNode 和 DataNode 的通用配置参数位于 `conf` 目录下。 | 名字 | mpp\_data\_exchange\_keep\_alive\_time\_in\_ms | |:------:|:-----------------------------------------------| | 描述 | MPP 数据交换最大等待时间 | -| 类型 | int | +| 类型 | Int32 | | 默认值 | 1000 | | 改后生效方式 | 重启服务生效 | -* default\_fill\_interval - -|名字| default\_fill\_interval | -|:---:|:---| -|描述| 填充查询中使用的默认时间段，默认-1表示无限过去时间，以毫秒ms为单位 | -|类型| Int32 | -|默认值| -1 | -|改后生效方式|重启服务生效| - -* group_by_fill_cache_size_in_mb - -| 名字 | group_by_fill_cache_size_in_mb | -| :----------: | :--------------------------------- | -| 描述 | 填充查询中使用的缓存大小，单位是MB | -| 类型 | Float | -| 默认值 | 1.0 | -| 改后生效方式 | 重启服务生效 | - * driver\_task\_execution\_time\_slice\_in\_ms |名字| driver\_task\_execution\_time\_slice\_in\_ms | |:---:|:---| |描述| 单个 DriverTask 最长执行时间 | -|类型| int | +|类型| Int32 | |默认值| 100 | |改后生效方式|重启服务生效| @@ -472,7 +463,7 @@ IoTDB ConfigNode 和 DataNode 的通用配置参数位于 `conf` 目录下。 |名字| max\_tsblock\_size\_in\_bytes | |:---:|:---| |描述| 单个 TsBlock 的最大容量 | -|类型| int | +|类型| Int32 | |默认值| 1024 * 1024 (1 MB) | |改后生效方式|重启服务生效| @@ -481,7 +472,7 @@ IoTDB ConfigNode 和 DataNode 的通用配置参数位于 `conf` 目录下。 |名字| max\_tsblock\_line\_numbers | |:---:|:---| |描述| 单个 TsBlock 的最大行数 | -|类型| int | +|类型| Int32 | |默认值| 1000 | |改后生效方式|重启服务生效| @@ -489,29 +480,47 @@ IoTDB ConfigNode 和 DataNode 的通用配置参数位于 `conf` 目录下。 |名字| slow\_query\_threshold | |:---:|:---| -|描述| 慢查询的时间成本（毫秒ms）阈值。 | +|描述| 慢查询的时间阈值。单位：毫秒。| |类型| Int32 | |默认值| 5000 | |改后生效方式|触发生效| -* enable\_external\_sort +* query\_timeout\_threshold -|名字| enable\_external\_sort | +|名字| query\_timeout\_threshold | |:---:|:---| -|描述| 是否开启外部排序功能 | -|类型| Boolean | -|默认值| true | -|改后生效方式|重启服务生效| +|描述| 查询的最大执行时间。单位：毫秒。| +|类型| Int32 | +|默认值| 60000 | +|改后生效方式| 重启服务生效| -* external\_sort\_threshold +* max\_allowed\_concurrent\_queries -|名字| external\_sort\_threshold | +|名字| max\_allowed\_concurrent\_queries | |:---:|:---| -|描述| 单个时间序列的最大同时块读取数。若同时chunk读取的数量大于external_sort_threshold，则使用外部排序。当external_sort_threshold增加时，内存中同时排序的chunk数量可能会增加，这会占用更多的内存；external_sort_threshold 减小时，触发外部排序会增加耗时。| +|描述| 允许的最大并发查询数量。 | |类型| Int32 | |默认值| 1000 | |改后生效方式|重启服务生效| +* query\_thread\_count + +|名字| query\_thread\_count | +|:---:|:----------------------------------------------------------------------------| +|描述| 当 IoTDB 对内存中的数据进行查询时，最多启动多少个线程来执行该操作。如果该值小于等于 0，那么采用机器所安装的 CPU 核的数量。 | +|类型| Int32 | +|默认值| CPU 核数 | +|改后生效方式| 重启服务生效 | + +* batch\_size + +|名字| batch\_size | +|:---:|:---| +|描述| 服务器中每次迭代的数据量（数据条目，即不同时间戳的数量。） | +|类型| Int32 | +|默认值| 100000 | +|改后生效方式|重启服务生效| + ### 存储引擎配置 * timestamp\_precision @@ -613,51 +622,6 @@ IoTDB ConfigNode 和 DataNode 的通用配置参数位于 `conf` 目录下。 |默认值| 0 | |改后生效方式|重启服务生效| -* query\_thread\_count - -|名字| query\_thread\_count | -|:---:|:----------------------------------------------------------------------------| -|描述| 当 IoTDB 对内存中的数据进行查询时，最多启动多少个线程来执行该操作。如果该值小于等于 0，那么采用机器所安装的 CPU 核的数量。默认值为 16。 | -|类型| Int32 | -|默认值| 16 | -|改后生效方式| 重启服务生效 | - -* sub\_rawQuery\_thread\_count - -|名字| sub\_rawQuery\_thread\_count | -|:---:|:------------------------------------------------| -|描述| 原始数据查询时，最多启动多少个线程来执行该操作。如果设置小于等于 0，会采用机器 CPU 核数 | -|类型| Int32 | -|默认值| 8 | -|改后生效方式| 重启服务生效 | - -* raw\_query\_blocking\_queue\_capacity - -|名字| raw\_query\_blocking\_queue\_capacity | -|:---:|:--------------------------------------| -|描述| 原始数据查询中，读任务的阻塞队列长度。默认值为 5 | -|类型| Int32 | -|默认值| 5 | -|改后生效方式| 重启服务生效 | - -* chunk\_buffer\_pool\_enable - -|名字| chunk\_buffer\_pool\_enable | -|:---:|:---| -|描述| 在将 memtable 序列化为内存中的字节时，是否开启由 IoTDB 而不是 JVM 接管内存管理，默认关闭。 | -|类型| Boolean | -|默认值| false | -|改后生效方式|重启服务生效| - -* batch\_size - -|名字| batch\_size | -|:---:|:---| -|描述| 服务器中每次迭代的数据量（数据条目，即不同时间戳的数量。） | -|类型| Int32 | -|默认值| 100000 | -|改后生效方式|重启服务生效| - * enable\_partial\_insert |名字| enable\_partial\_insert | @@ -831,15 +795,6 @@ IoTDB ConfigNode 和 DataNode 的通用配置参数位于 `conf` 目录下。 |默认值| 16 | |改后生效方式| 重启服务生效| -* query\_timeout\_threshold - -|名字| query\_timeout\_threshold | -|:---:|:---| -|描述| 查询的最大执行时间。单位：毫秒。| -|类型| Int32 | -|默认值| 60000 | -|改后生效方式| 重启服务生效| - ### 写前日志配置 * wal\_buffer\_size @@ -961,44 +916,6 @@ IoTDB ConfigNode 和 DataNode 的通用配置参数位于 `conf` 目录下。 |默认值| 1024 | |改后生效方式|触发生效| -### 水印模块配置 - -* watermark\_module\_opened - -|名字| watermark\_module\_opened | -|:---:|:---| -|描述| 是否开启水印水印嵌入功能 | -|取值| true or false | -|默认值| false | -|改后生效方式|重启服务生效| - -* watermark\_secret\_key - -|名字| watermark\_secret\_key | -|:---:|:---| -|描述| 水印嵌入功能秘钥 | -|类型| String | -|默认值| IoTDB * 2019@Beijing | -|改后生效方式|重启服务生效| - -* watermark\_bit\_string - -|名字| watermark\_bit\_string | -|:---:|:---| -|描述| 水印比特字符串 | -|类型| Int32 | -|默认值| 100101110100 | -|改后生效方式|重启服务生效| - -* watermark\_method - -|名字| watermark\_method | -|:---:|:---| -|描述| 水印嵌入方法 | -|类型| String | -|默认值| GroupBasedLSBMethod(embed_row_cycle=2,embed_lsb_num=5) | -|改后生效方式|重启服务生效| - ### 授权配置 * authorizer\_provider\_class diff --git a/node-commons/src/assembly/resources/conf/iotdb-common.properties b/node-commons/src/assembly/resources/conf/iotdb-common.properties index 21138fcbc2..7863a5d749 100644 --- a/node-commons/src/assembly/resources/conf/iotdb-common.properties +++ b/node-commons/src/assembly/resources/conf/iotdb-common.properties @@ -327,7 +327,7 @@ # Datatype: string # read_consistency_level=strong -# whether to cache meta data(BloomFilter, ChunkMetadata and TimeSeriesMetadata) or not. +# Whether to cache meta data(BloomFilter, ChunkMetadata and TimeSeriesMetadata) or not. # Datatype: boolean # meta_data_cache_enable=true @@ -339,53 +339,54 @@ # Datatype: boolean # enable_last_cache=true -# allowed max numbers of deduplicated path in one query +# Allowed max numbers of deduplicated path in one query # it's just an advised value, the real limitation will be the smaller one between this and the one we calculated # Datatype: int # max_deduplicated_path_num=1000 +# Core size of ThreadPool of MPP data exchange # Datatype: int # mpp_data_exchange_core_pool_size=10 +# Max size of ThreadPool of MPP data exchange # Datatype: int # mpp_data_exchange_max_pool_size=10 +# Max waiting time for MPP data exchange # Datatype: int # mpp_data_exchange_keep_alive_time_in_ms=1000 -# the default time period that used in fill query, -1 by default means infinite past time -# Datatype: int, Unit: ms -# default_fill_interval=-1 - -# Datatype: float -# group_by_fill_cache_size_in_mb=1.0 - -# the max execution time of a DriverTask +# The max execution time of a DriverTask # Datatype: int, Unit: ms # driver_task_execution_time_slice_in_ms=100 -# the max capacity of a TsBlock +# The max capacity of a TsBlock # Datatype: int, Unit: byte # max_tsblock_size_in_bytes=1048576 -# the max number of lines in a single TsBlock +# The max number of lines in a single TsBlock # Datatype: int # max_tsblock_line_numbers=1000 -# time cost(ms) threshold for slow query +# Time cost(ms) threshold for slow query # Datatype: long # slow_query_threshold=5000 -# Is external sort enable -# Datatype: boolean -# enable_external_sort=true +# The max executing time of query. unit: ms +# Datatype: int +# query_timeout_threshold=60000 + +# The maximum allowed concurrently executing queries +# Datatype: int +# max_allowed_concurrent_queries=1000 + +# How many threads can concurrently execute query statement. When <= 0, use CPU core number. +# Datatype: int +# query_thread_count=0 -# The maximum number of simultaneous chunk reading for a single time series. -# If the num of simultaneous chunk reading is greater than external_sort_threshold, external sorting is used. -# When external_sort_threshold increases, the number of chunks sorted at the same time in memory may increase and this will occupy more memory. -# When external_sort_threshold decreases, triggering external sorting will increase the time-consuming. +# The amount of data iterate each time in server (the number of data strips, that is, the number of different timestamps.) # Datatype: int -# external_sort_threshold=1000 +# batch_size=100000 #################### ### Storage Engine Configuration @@ -470,27 +471,6 @@ # Datatype: int # flush_thread_count=0 -# How many threads can concurrently execute query statement. When <= 0, use CPU core number. -# Datatype: int -# query_thread_count=0 - -# How many threads can concurrently read data for raw data query. When <= 0, use CPU core number. -# Datatype: int -# sub_rawQuery_thread_count=8 - -# Blocking queue size for read task in raw data query. Must >= 1. -# Datatype: int -# raw_query_blocking_queue_capacity=5 - -# whether take over the memory management by IoTDB rather than JVM when serializing memtable as bytes in memory -# (i.e., whether use ChunkBufferPool), value true, false -# Datatype: boolean -# chunk_buffer_pool_enable=false - -# The amount of data iterate each time in server (the number of data strips, that is, the number of different timestamps.) -# Datatype: int -# batch_size=100000 - # In one insert (one device, one timestamp, multiple measurements), # if enable partial insert, one measurement failure will not impact other measurements # Datatype: boolean @@ -615,14 +595,6 @@ # Datatype: int # compaction_write_throughput_mb_per_sec=16 -# The max executing time of query. unit: ms -# Datatype: int -# query_timeout_threshold=60000 - -# The maximum allowed concurrently executing queries -# Datatype: int -# max_allowed_concurrent_queries=1000 - # The number of sub compaction threads to be set up to perform compaction. # Currently only works for nonAligned data in cross space compaction and unseq inner space compaction. # Set to 1 when less than or equal to 0.
