This is an automated email from the ASF dual-hosted git repository. kxiao pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git
commit 79a5663a7d7637cdc367f0f3e387d97d80e65250 Author: zhengyu <[email protected]> AuthorDate: Tue Jul 11 19:52:14 2023 +0800 [fix](config) set FE header size limit to 1MB from 10k (#21719) Enlarge jetty_server_max_http_header_size to avoid Request Header Fields Too Large error when streamloading to FE. Signed-off-by: freemandealer <[email protected]> --- docs/en/docs/admin-manual/config/fe-config.md | 37 +++++++++++----------- docs/zh-CN/docs/admin-manual/config/fe-config.md | 22 ++++++------- .../main/java/org/apache/doris/common/Config.java | 6 ++-- 3 files changed, 32 insertions(+), 33 deletions(-) diff --git a/docs/en/docs/admin-manual/config/fe-config.md b/docs/en/docs/admin-manual/config/fe-config.md index f40fce369b..65911bf27a 100644 --- a/docs/en/docs/admin-manual/config/fe-config.md +++ b/docs/en/docs/admin-manual/config/fe-config.md @@ -7,7 +7,7 @@ } --- -<!-- +<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information @@ -82,7 +82,7 @@ There are two ways to configure FE configuration items: **Configuration items modified in this way will become invalid after the FE process restarts.** For more help on this command, you can view it through the `HELP ADMIN SET CONFIG;` command. - + 3. Dynamic configuration via HTTP protocol For details, please refer to [Set Config Action](../http-actions/fe/set-config-action.md) @@ -335,7 +335,7 @@ The multi cluster feature will be deprecated in version 0.12 ，set this config Default：disable -Set to true if you deploy Doris using thirdparty deploy manager +Set to true if you deploy Doris using thirdparty deploy manager Valid options are: @@ -386,11 +386,11 @@ FE MySQL server port #### `frontend_address` -Status: Deprecated, not recommended use. This parameter may be deleted later +Status: Deprecated, not recommended use. This parameter may be deleted later -Type: string +Type: string -Description: Explicitly set the IP address of FE instead of using *InetAddress.getByName* to get the IP address. Usually in *InetAddress.getByName* When the expected results cannot be obtained. Only IP address is supported, not hostname. +Description: Explicitly set the IP address of FE instead of using *InetAddress.getByName* to get the IP address. Usually in *InetAddress.getByName* When the expected results cannot be obtained. Only IP address is supported, not hostname. Default value: 0.0.0.0 @@ -633,7 +633,7 @@ HTTP Server V2 is implemented by SpringBoot. It uses an architecture that separa #### `http_api_extra_base_path` -In some deployment environments, user need to specify an additional base path as the unified prefix of the HTTP API. This parameter is used by the user to specify additional prefixes. +In some deployment environments, user need to specify an additional base path as the unified prefix of the HTTP API. This parameter is used by the user to specify additional prefixes. After setting, user can get the parameter value through the `GET /api/basepath` interface. And the new UI will also try to get this base path first to assemble the URL. Only valid when `enable_http_server_v2` is true. The default is empty, that is, not set @@ -664,7 +664,7 @@ This is the maximum number of bytes of the file uploaded by the put or post meth #### `jetty_server_max_http_header_size` -Default：10240 （10K） +Default：1048576 （1M） http header size configuration parameter, the default value is 10K @@ -907,7 +907,7 @@ Default：false IsMutable：true -If set to true, Planner will try to select replica of tablet on same host as this Frontend. +If set to true, Planner will try to select replica of tablet on same host as this Frontend. This may reduce network transmission in following case: - N hosts with N Backends and N Frontends deployed. @@ -1070,8 +1070,8 @@ MasterOnly：true if this is set to true -- all pending load job will failed when call begin txn api -- all prepare load job will failed when call commit txn api +- all pending load job will failed when call begin txn api +- all prepare load job will failed when call commit txn api - all committed load job will waiting to be published #### `commit_timeout_second` @@ -1488,9 +1488,9 @@ IsMutable：true MasterOnly：true -labels of finished or cancelled load jobs will be removed after `label_keep_max_second` ， +labels of finished or cancelled load jobs will be removed after `label_keep_max_second` ， -1. The removed labels can be reused. +1. The removed labels can be reused. 2. Set a short time will lower the FE memory usage. (Because all load jobs' info is kept in memory before being removed) In the case of high concurrent writes, if there is a large backlog of jobs and call frontend service failed, check the log. If the metadata write takes too long to lock, you can adjust this value to 12 hours, or 6 hours less @@ -2081,7 +2081,7 @@ IsMutable：true MasterOnly：true -the factor of delay time before deciding to repair tablet. +the factor of delay time before deciding to repair tablet. - if priority is VERY_HIGH, repair it immediately. - HIGH, delay tablet_repair_delay_factor_second * 1; @@ -2160,8 +2160,7 @@ When create a table(or partition), you can specify its storage medium(HDD or SSD #### `enable_storage_policy` -Whether to enable the Storage Policy feature. This feature allows users to separate hot and cold data. This feature is still under development. Recommended for test environments only. - +- Whether to enable the Storage Policy feature. This config allows users to separate hot and cold data. Default: false Is it possible to dynamically configure: true @@ -2310,7 +2309,7 @@ IsMutable：true MasterOnly：true -Whether to enable the ODBC table, it is not enabled by default. You need to manually configure it when you use it. +Whether to enable the ODBC table, it is not enabled by default. You need to manually configure it when you use it. This parameter can be set by: ADMIN SET FRONTEND CONFIG("key"="value") @@ -2738,5 +2737,5 @@ This is used to limit the setting of "parallel_fragment_exec_instance_num". Default: 4 -This variable indicates the number of digits by which to increase the scale of the result of -division operations performed with the `/` operator. +This variable indicates the number of digits by which to increase the scale of the result of +division operations performed with the `/` operator. diff --git a/docs/zh-CN/docs/admin-manual/config/fe-config.md b/docs/zh-CN/docs/admin-manual/config/fe-config.md index 0bd7409d1e..22b9c66032 100644 --- a/docs/zh-CN/docs/admin-manual/config/fe-config.md +++ b/docs/zh-CN/docs/admin-manual/config/fe-config.md @@ -8,7 +8,7 @@ --- -<!-- +<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information @@ -231,7 +231,7 @@ Master FE 的 bdbje 同步策略。 如果您只部署一个 Follower FE，请 是否可以动态配置：true -如果为 true，非主 FE 将忽略主 FE 与其自身之间的元数据延迟间隙，即使元数据延迟间隙超过 `meta_delay_toleration_second`。 +如果为 true，非主 FE 将忽略主 FE 与其自身之间的元数据延迟间隙，即使元数据延迟间隙超过 `meta_delay_toleration_second`。 非主 FE 仍将提供读取服务。 当您出于某种原因尝试停止 Master FE 较长时间，但仍希望非 Master FE 可以提供读取服务时，这会很有帮助。 #### `meta_delay_toleration_second` @@ -388,7 +388,7 @@ Doris FE 通过 mysql 协议查询连接端口 状态:已弃用，不建议使用。 -类型:string +类型:string 描述:显式配置FE的IP地址，不使用*InetAddress。getByName*获取IP地址。通常在*InetAddress中。getByName*当无法获得预期结果时。只支持IP地址，不支持主机名。 @@ -477,7 +477,7 @@ thrift 服务器的 backlog_num 当你扩大这个 backlog_num 时，你应该 默认值：0 -thrift 服务器的连接超时和套接字超时配置 +thrift 服务器的连接超时和套接字超时配置 thrift_client_timeout_ms 的默认值设置为零以防止读取超时 @@ -664,7 +664,7 @@ workers 线程池默认不做设置，根据自己需要进行设置 #### `jetty_server_max_http_header_size` -默认值：10240 （10K） +默认值：1048576 （1M） http header size 配置参数 @@ -796,8 +796,8 @@ trace导出到 collector: `http://127.0.0.1:4318/v1/traces` 用于控制用户表表名大小写是否敏感。 该配置只能在集群初始化时配置，初始化完成后集群重启和升级时不能修改。 -0：表名按指定存储，比较区分大小写。 -1：表名以小写形式存储，比较不区分大小写。 +0：表名按指定存储，比较区分大小写。 +1：表名以小写形式存储，比较不区分大小写。 2：表名按指定存储，但以小写形式进行比较。 #### `table_name_length_limit` @@ -1471,7 +1471,7 @@ NORMAL 优先级挂起加载作业的并发数。 负载中落后节点的最大等待秒数 例如： - 有 3 个副本 A, B, C + 有 3 个副本 A, B, C load 已经在 t1 时仲裁完成 (A,B) 并且 C 没有完成， 如果 (current_time-t1)> 300s，那么 doris会将 C 视为故障节点， 将调用事务管理器提交事务并告诉事务管理器 C 失败。 @@ -1781,7 +1781,7 @@ show data （其他用法：HELP SHOW DATA） 是否为 Master FE 节点独有的配置项：true -在某些情况下，某些 tablet 可能会损坏或丢失所有副本。 此时数据已经丢失，损坏的 tablet 会导致整个查询失败，无法查询剩余的健康 tablet。 +在某些情况下，某些 tablet 可能会损坏或丢失所有副本。 此时数据已经丢失，损坏的 tablet 会导致整个查询失败，无法查询剩余的健康 tablet。 在这种情况下，您可以将此配置设置为 true。 系统会将损坏的 tablet 替换为空 tablet，以确保查询可以执行。 （但此时数据已经丢失，所以查询结果可能不准确） @@ -1800,7 +1800,7 @@ show data （其他用法：HELP SHOW DATA） 这个配置有三个值： * disable ：出现异常会正常报错。 - * ignore_version: 忽略 fe partition 中记录的visibleVersion 信息， 使用replica version + * ignore_version: 忽略 fe partition 中记录的visibleVersion 信息， 使用replica version * ignore_all: 除了ignore_version， 在遇到找不到可查询的replica 时，直接跳过而不是抛出异常 #### `min_clone_task_timeout_sec` 和 `max_clone_task_timeout_sec` @@ -2567,7 +2567,7 @@ SmallFileMgr 中存储的最大文件数 是否为 Master FE 节点独有的配置项：true -这个阈值是为了避免在 FE 中堆积过多的报告任务，可能会导致 OOM 异常等问题。 +这个阈值是为了避免在 FE 中堆积过多的报告任务，可能会导致 OOM 异常等问题。 并且每个 BE 每 1 分钟会报告一次 tablet 信息，因此无限制接收报告是不可接受的。 以后我们会优化 tablet 报告的处理速度 diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java index d3e8298372..a7af0a0c3b 100644 --- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java +++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java @@ -361,9 +361,9 @@ public class Config extends ConfigBase { "The maximum HTTP POST size of Jetty, in bytes, the default value is 100MB."}) public static int jetty_server_max_http_post_size = 100 * 1024 * 1024; - @ConfField(description = {"Jetty 的最大 HTTP header 大小，单位是字节，默认值是 10KB。", - "The maximum HTTP header size of Jetty, in bytes, the default value is 10KB."}) - public static int jetty_server_max_http_header_size = 10240; + @ConfField(description = {"Jetty 的最大 HTTP header 大小，单位是字节，默认值是 1MB。", + "The maximum HTTP header size of Jetty, in bytes, the default value is 1MB."}) + public static int jetty_server_max_http_header_size = 1048576; @ConfField(description = {"是否禁用 mini load，默认禁用", "Whether to disable mini load, disabled by default"}) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
