foxtail463 commented on PR #67817: URL: https://github.com/apache/doris/pull/67817#issuecomment-5760884402
<h1 data-line="0" class="code-line code-active-line" dir="auto" id="array-rowid-批读--benchmark-报告" style="margin-top: 0px; font-weight: 600; margin-bottom: 16px; line-height: 1.25; font-size: 2em; padding-bottom: 0.3em; border-bottom: 1px solid rgb(0, 0, 0); border-top-color: rgb(0, 0, 0); border-right-color: rgb(0, 0, 0); border-left-color: rgb(0, 0, 0); color: rgb(255, 255, 255); font-family: -apple-system, "system-ui", "Segoe WPC", "Segoe UI", system-ui, Ubuntu, "Droid Sans", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">ARRAY RowID 批读 · Benchmark 报告</h1><blockquote data-line="2" class="code-line" dir="au to" style="background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0); border-color: rgb(255, 255, 255); margin: 0px; padding: 0px 16px 0px 10px; border-left-width: 5px; border-left-style: solid; border-radius: 2px; color: rgb(255, 255, 255); font-family: -apple-system, "system-ui", "Segoe WPC", "Segoe UI", system-ui, Ubuntu, "Droid Sans", sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><p data-line="2" class="code-line" dir="auto" style="margin-top: 0px; margin-bottom: 16px;">完整图表见同目录<span> </span><code style="font-family: Menlo, Monaco, "Courier New", monospace; color: rgb(255, 255, 255); background-color: rgba(0, 0, 0, 0); padding: 1px 3px; border-radius: 4px; font-size: 1em; line-height: 1.357em;">report.html</code>，精确数值见<span> </span><code style="font-family: Menlo, Monaco, "Courier New", monospace; color: rgb(255, 255, 255); background-color: rgba(0, 0, 0, 0); padding: 1px 3px; border-radius: 4px; font-size: 1em; line-height: 1.357em;">*.csv</code>。本文为其 Markdown 摘要，只比较<strong>旧逐行实现（L）<strong>与</strong>本 PR 批读实现（B）</strong>；实验中还存在中间方案（父连续段合并），本文不展开。</p></blockquote><h2 data-line="4" class="code-line" dir="auto" id="概述" style="margin-top: 24px; font-weight: 600; margin-bottom: 16px; line-height: 1.25; font-size: 1.5em; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; color: rgb(255, 255, 255); font-family: -apple-system, "system-ui", "Segoe WPC& quot;, "Segoe UI", system-ui, Ubuntu, "Droid Sans", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">概述</h2><ul data-line="6" class="code-line" dir="auto" style="margin-top: 0px; margin-bottom: 0.7em; color: rgb(255, 255, 255); font-family: -apple-system, "system-ui", "Segoe WPC", "Segoe UI", system-ui, Ubuntu, "Droid Sans", sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; wh ite-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><li data-line="6" class="code-line" dir="auto"><strong>测试对象</strong>：同一 RELEASE 基线，只改变<span> </span><code style="font-family: Menlo, Monaco, "Courier New", monospace; color: rgb(255, 255, 255); background-color: rgba(0, 0, 0, 0); padding: 1px 3px; border-radius: 4px; font-size: 1em; line-height: 1.357em;">ArrayFileColumnIterator::read_by_rowids</code>。基线 commit：<code style="font-family: Menlo, Monaco, "Courier New", monospace; color: rgb(255, 255, 255); background-color: rgba(0, 0, 0, 0); padding: 1px 3px; border-radius: 4px; font-size: 1em; line-height: 1.357em;">b78724a75f704b059bf07cab4e7fa6601bdd8a79</code>。</li><li data-line="7" class="code-line" dir="auto"><strong>两个方案</strong>：<ul data-line="8" class="code-line" dir="auto" style="margin-top: 0px; margin-bottom: 0.7em;"><li data-line="8" class ="code-line" dir="auto"><strong>L · 逐行（旧代码）</strong>：每个父 RowID 执行一次 seek +<span> </span><code style="font-family: Menlo, Monaco, "Courier New", monospace; color: rgb(255, 255, 255); background-color: rgba(0, 0, 0, 0); padding: 1px 3px; border-radius: 4px; font-size: 1em; line-height: 1.357em;">next_batch(1)</code></li><li data-line="9" class="code-line" dir="auto"><strong>B · 批读（本 PR）</strong>：批读 NULL / offset，再合并物理 item 区间</li></ul></li><li data-line="10" class="code-line" dir="auto"><strong>数据</strong>：确定性合成标签，非生产回放。每表 33,554,432 行、8 tablet、单副本。普通短 ARRAY：父 NULL 约 10%、empty 约 5%、非空长度 5–10。</li><li data-line="11" class="code-line" dir="auto"><strong>并发 SQL</strong>：全扫大表 → 标量条件留 3,691,952 行 → 读三个 ARRAY 做标签过滤，最终 104,016 行。</li></ul><h2 data-line="13" class="code-line" dir="au to" id="环境与方法" style="margin-top: 24px; font-weight: 600; margin-bottom: 16px; line-height: 1.25; font-size: 1.5em; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; color: rgb(255, 255, 255); font-family: -apple-system, "system-ui", "Segoe WPC", "Segoe UI", system-ui, Ubuntu, "Droid Sans", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">环境与方法</h2><p data-line="15" class="code-line" dir="auto" style="margin-top: 0px; margin-bottom: 16px; color: rgb(255, 255, 255); font-family: -apple-system, "system-ui", "Segoe WPC", "Segoe UI", system-ui, Ubuntu, &quo t;Droid Sans", sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">单 FE / 单 BE。BE 亲和 8 个物理核（非独占）、32 GiB 内存上限、8 GiB 页缓存、pipeline=4、batch_size=4096。关闭 SQL/query/condition cache、自动 compaction 和自动统计。每版本切换后重启 BE 并预热，正式查询页缓存命中 100%。</p><p data-line="17" class="code-line" dir="auto" style="margin-top: 0px; margin-bottom: 16px; color: rgb(255, 255, 255); font-family: -apple-system, "system-ui", "Segoe WPC", "Segoe UI", system-ui, Ubuntu, "Droid Sans", sans-serif; font-size: 14px; font -style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">并发 1 / 8 / 32 / 64 / 128，各档样本 16 / 16 / 64 / 128 / 256。<strong>总计 1440 条正式 SQL、72 个批次</strong>（实验按 L → R → B → B → R → L 正反顺序执行以降低顺序偏差，本摘要只呈现 L 与 B）。</p><ul data-line="19" class="code-line" dir="auto" style="margin-top: 0px; margin-bottom: 0.7em; color: rgb(255, 255, 255); font-family: -apple-system, "system-ui", "Segoe WPC", "Segoe UI", system-ui, Ubuntu, "Droid Sans", sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-w eight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><li data-line="19" class="code-line" dir="auto">吞吐 = 完成查询数 ÷ 批次执行时间（非开放负载稳态 QPS）；计时不含连接建立 / 查询 ID 查找 / Profile 下载。</li><li data-line="20" class="code-line" dir="auto">CPU 为整批 BE 进程 CPU 增量 ÷ 查询数。</li><li data-line="21" class="code-line" dir="auto">RSS 每 20ms 采样；单查询内存取 QueryContext 析构日志的 MemTracker 峰值（有批量记账和精度限制），各查询峰值不相加。</li></ul><h2 data-line="23" class="code-line" dir="auto" id="并发主实验2026-09-20" style="margin-top: 24px; font-weight: 600; margin-bottom: 16px; line-height: 1.25; font-size: 1.5em; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; color: rgb(255, 255, 255); font-family: -apple-system, "system-ui", "Segoe WPC", "Segoe UI", system-ui, Ubuntu, "Droid Sans", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">并发主实验（2026-09-20）</h2><h3 data-line="25" class="code-line" dir="auto" id="各档并发--吞吐与-cpu" style="margin-top: 24px; font-weight: 600; margin-bottom: 16px; line-height: 1.25; font-size: 1.25em; color: rgb(255, 255, 255); font-family: -apple-system, "system-ui", "Segoe WPC", "Segoe UI", system-ui, Ubuntu, "Droid Sans", sans-serif; fon t-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">各档并发 · 吞吐与 CPU</h3> 并发 | 逐行 q/s | 批读 q/s | 逐行 CPU秒/SQL | 批读 CPU秒/SQL -- | -- | -- | -- | -- 1 | 0.345 | 0.542 | 11.02 | 6.91 8 | 0.731 | 1.098 | 10.67 | 6.96 32 | 0.730 | 1.157 | 10.89 | 6.86 64 | 0.750 | 1.170 | 10.59 | 6.79 128 | 0.744 | 1.176 | 10.69 | 6.77 <p data-line="80" class="code-line" dir="auto" style="margin-top: 0px; margin-bottom: 16px; color: rgb(255, 255, 255); font-family: -apple-system, "system-ui", "Segoe WPC", "Segoe UI", system-ui, Ubuntu, "Droid Sans", sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">所有测试场景中批读均不低于逐行；没有发现相对旧代码的回退点。小批量的组织开销体现在与中间方案的对比中（16 个连续父行的小批，批读比简单连续段合并慢约 16%），但相对旧逐行实现仍净收益。</p><h2 data-line="82" class="code-line" dir="auto" id="结论" style="margin-top: 24px; font-weight: 600; margin-bottom: 16px; line-height: 1.25; font-size: 1.5em; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; color: rgb(255, 255, 255); font-family: -apple-system, "system-ui", "Segoe WPC", "Segoe UI", system-ui, Ubuntu, "Droid Sans", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">结论</h2><ul data-line="84" class="code-line" dir="auto" style="margin-top: 0px; margin-bottom: 0.7em; color: rgb(255, 255, 255); font-family: -apple-system, "system-ui", "Segoe WPC", "Segoe UI", system-ui, Ubuntu, "Droid Sans", sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><li data-line="84" class="code-line" dir="auto">在大量离散 ARRAY 读取负载下，批读较旧逐行实现吞吐 +50%～+58%、CPU/SQL −35%～−37%。</li><li data-line="85" class="code-line" dir="auto">高并发存在可测的内存增长（64 并发 +125 MiB、128 并发 +229 MiB 进程峰值，运行增量 +15.1%）。</li><li data-line="86" class="code-line" dir="auto">支持稀疏读取场景的优化价值；不应宣传为零资源代价或所有 SQL 普遍加速（顺序 / 聚合主导的 SQL 持平）。</li></ul><h2 data-line="88" class="code-line" dir="auto" id="结果可靠性 与外推限制" style="margin-top: 24px; font-weight: 600; margin-bottom: 16px; line-height: 1.25; font-size: 1.5em; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; color: rgb(255, 255, 255); font-family: -apple-system, "system-ui", "Segoe WPC", "Segoe UI", system-ui, Ubuntu, "Droid Sans", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">结果可靠性与外推限制</h2><ul data-line="90" class="code-line" dir="auto" style="margin-top: 0px; margin-bottom: 0.7em; color: rgb(255, 255, 255); font-family: -apple-system, "system-ui", "Segoe WPC", "Segoe UI", system-ui, Ubu ntu, "Droid Sans", sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><li data-line="90" class="code-line" dir="auto">正式 1440 条 SQL 零错误，结果与扫描 / 两阶段过滤行数一致；ConditionCache 命中与过滤均为 0。</li><li data-line="91" class="code-line" dir="auto">本轮并发只覆盖随机 11%、三个短字符串 ARRAY；其他业务分布、冷热缓存与硬件需另行验证。</li><li data-line="92" class="code-line" dir="auto">CPU 亲和非独占，主机存在其他负载；正反顺序复测降低顺序偏差，但不构成生产性能或统计显著性保证。</li><li data-line="93" clas s="code-line" dir="auto">RSS 为采样峰值；MemTracker 有批量记账与日志精度限制；高并发每档仅两次独立批次，P95 不代表生产 SLO。</li></ul> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
