leaves12138 commented on code in PR #553: URL: https://github.com/apache/paimon-rust/pull/553#discussion_r3612867904
########## docs/src/html/benchmark/index.html: ########## @@ -0,0 +1,434 @@ +<!DOCTYPE html> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> +<html lang="en"> +<head> +<meta charSet="utf-8"/> +<meta name="viewport" content="width=device-width, initial-scale=1"/> +<link rel="stylesheet" href="styles.css"/> +<title>Paimon Rust Reaches a Performance Turning Point | TPC-DS SF100</title> +<meta name="description" content="On DataFusion 54, Paimon Rust finishes the comparable TPC-DS SF100 queries within 0.27% of native Parquet, while the heavy-query group is 12.9% faster."/> +<meta property="og:title" content="Paimon Rust Reaches a Performance Turning Point"/> +<meta property="og:description" content="TPC-DS SF100: total workload execution time is within 0.27% of DataFusion native Parquet."/> +<meta property="og:type" content="article"/> +<meta property="og:locale" content="en_US"/> +<meta name="twitter:card" content="summary"/> +<meta name="twitter:title" content="Paimon Rust Reaches a Performance Turning Point"/> +<meta name="twitter:description" content="TPC-DS SF100: total workload execution time is within 0.27% of DataFusion native Parquet."/> +</head> +<body> +<main> +<script type="application/ld+json">{"@context":"https://schema.org","@type":"TechArticle","headline":"Paimon Rust Reaches a Performance Turning Point: TPC-DS SF100 Parity with Native Parquet on DataFusion","datePublished":"2026-07-20","inLanguage":"en","author":{"@type":"Organization","name":"Paimon Rust Community"},"description":"Paimon Rust finishes the comparable TPC-DS SF100 queries within 0.27% of DataFusion native Parquet."}</script> +<header class="topbar"> +<a class="brand" href="#top" aria-label="Back to the top"> +<span class="brand-mark">P</span> +<span>PAIMON RUST</span> +</a> +<nav aria-label="Article navigation"> +<a href="#result">Results</a> +<a href="#engineering">Engineering</a> +<a href="#ecosystem">Ecosystem</a> +<a href="#method">Methodology</a> +</nav> +<a class="github-link" href="https://github.com/apache/paimon-rust"; target="_blank" rel="noreferrer">GitHub ↗</a> +</header> +<article id="top"> +<section class="hero section-shell"> +<div class="hero-glow" aria-hidden="true"> +</div> +<div class="eyebrow"> +<span> +</span> PERFORMANCE MILESTONE · JUL 2026</div> +<h1><strong>Paimon Rust</strong> <em>At Native Speed</em> +</h1> +<p class="hero-lead">In a TPC-DS SF100 workload on DataFusion 54, Paimon Rust finishes the 96 comparable queries <strong>within 0.27% of native Parquet</strong>. For the queries that dominate execution time, Paimon is 12.9% faster in aggregate. Table semantics, snapshots, schema evolution, and lakehouse capabilities no longer need to come with a material analytical-performance penalty.</p> Review Comment: Blocking accuracy issue: this benchmark harness imports the data into unpartitioned append-only Paimon tables (`benchmarks/tpcds/README.md`), so the run does not exercise primary-key merging, schema evolution, deletion vectors, row lineage, or multiple file formats. This sentence currently presents the result as evidence that those semantics carry no material performance penalty. Please scope the conclusion to the append-only SF100 read path tested here, and describe the broader features as capabilities rather than benchmarked outcomes. ########## docs/src/html/benchmark/index.html: ########## @@ -0,0 +1,434 @@ +<!DOCTYPE html> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> +<html lang="en"> +<head> +<meta charSet="utf-8"/> +<meta name="viewport" content="width=device-width, initial-scale=1"/> +<link rel="stylesheet" href="styles.css"/> +<title>Paimon Rust Reaches a Performance Turning Point | TPC-DS SF100</title> +<meta name="description" content="On DataFusion 54, Paimon Rust finishes the comparable TPC-DS SF100 queries within 0.27% of native Parquet, while the heavy-query group is 12.9% faster."/> +<meta property="og:title" content="Paimon Rust Reaches a Performance Turning Point"/> +<meta property="og:description" content="TPC-DS SF100: total workload execution time is within 0.27% of DataFusion native Parquet."/> +<meta property="og:type" content="article"/> +<meta property="og:locale" content="en_US"/> +<meta name="twitter:card" content="summary"/> +<meta name="twitter:title" content="Paimon Rust Reaches a Performance Turning Point"/> +<meta name="twitter:description" content="TPC-DS SF100: total workload execution time is within 0.27% of DataFusion native Parquet."/> +</head> +<body> +<main> +<script type="application/ld+json">{"@context":"https://schema.org","@type":"TechArticle","headline":"Paimon Rust Reaches a Performance Turning Point: TPC-DS SF100 Parity with Native Parquet on DataFusion","datePublished":"2026-07-20","inLanguage":"en","author":{"@type":"Organization","name":"Paimon Rust Community"},"description":"Paimon Rust finishes the comparable TPC-DS SF100 queries within 0.27% of DataFusion native Parquet."}</script> +<header class="topbar"> +<a class="brand" href="#top" aria-label="Back to the top"> +<span class="brand-mark">P</span> +<span>PAIMON RUST</span> +</a> +<nav aria-label="Article navigation"> +<a href="#result">Results</a> +<a href="#engineering">Engineering</a> +<a href="#ecosystem">Ecosystem</a> +<a href="#method">Methodology</a> +</nav> +<a class="github-link" href="https://github.com/apache/paimon-rust"; target="_blank" rel="noreferrer">GitHub ↗</a> +</header> +<article id="top"> +<section class="hero section-shell"> +<div class="hero-glow" aria-hidden="true"> +</div> +<div class="eyebrow"> +<span> +</span> PERFORMANCE MILESTONE · JUL 2026</div> +<h1><strong>Paimon Rust</strong> <em>At Native Speed</em> +</h1> +<p class="hero-lead">In a TPC-DS SF100 workload on DataFusion 54, Paimon Rust finishes the 96 comparable queries <strong>within 0.27% of native Parquet</strong>. For the queries that dominate execution time, Paimon is 12.9% faster in aggregate. Table semantics, snapshots, schema evolution, and lakehouse capabilities no longer need to come with a material analytical-performance penalty.</p> +<div class="hero-actions"> +<a class="button primary" href="#result">View Results</a> +<a class="button secondary" href="https://github.com/apache/paimon-rust/pull/549"; target="_blank" rel="noreferrer">Read the Implementation PR ↗</a> +</div> +<div class="metric-strip" role="list" aria-label="Key performance results"> +<div class="metric" role="listitem"> +<span class="metric-value">+0.27%</span> +<span class="metric-label">Total workload time gap</span> +</div> +<div class="metric" role="listitem"> +<span class="metric-value">30 / 96</span> +<span class="metric-label">Queries won by Paimon</span> +</div> +<div class="metric" role="listitem"> +<span class="metric-value accent">−12.9%</span> +<span class="metric-label">Aggregate heavy-query time</span> +</div> +<div class="metric" role="listitem"> +<span class="metric-value">5.4×</span> +<span class="metric-label">Q71 speedup</span> +</div> +</div> +</section> +<section class="statement section-shell"> +<p class="kicker">THE SHORT VERSION</p> +<blockquote>We are no longer asking whether Paimon in Rust can run.<br/>The question now is: <mark>how far can it go?</mark> +</blockquote> +</section> +<section class="results section-shell" id="result"> +<div class="section-heading"> +<div> +<p class="kicker">01 · THE RESULT</p> +<h2>The Same DataFusion.<br/>Paimon Now Stands Beside Parquet.</h2> +</div> +<p>This benchmark uses the same DataFusion SQL, planner, and execution operators to compare a Paimon source with DataFusion native Parquet. Because importing into Paimon rewrites the physical files, this is an end-to-end source comparison rather than a measurement of catalog overhead alone.</p> +</div> +<div class="score-card"> +<div class="score-head"> +<div> +<span class="score-label">96 queries successful on both sources · total execution time</span> +<strong>1.39 seconds</strong> +<small>Absolute difference across the entire workload</small> +</div> +<span class="parity-pill">PARITY REACHED</span> +</div> +<div class="bar-chart" aria-label="Total execution time: Paimon 516.44 seconds, Parquet 515.05 seconds"> +<div class="bar-row"> +<div class="bar-meta"> +<span>Paimon Rust</span> +<strong>516.44s</strong> +</div> +<div class="bar-track"> +<div class="bar paimon" style="width:100%"> +</div> +</div> +</div> +<div class="bar-row"> +<div class="bar-meta"> +<span>Native Parquet</span> +<strong>515.05s</strong> +</div> +<div class="bar-track"> +<div class="bar parquet" style="width:99.73%"> +</div> +</div> +</div> +</div> +<p class="score-note">“Parity” describes the aggregate workload, not identical performance on every query. Paimon still pays fixed overhead on some short queries. Among the 23 heavy queries for which Parquet takes at least five seconds, however, Paimon finishes in 339.86 seconds versus 390.00 seconds for native Parquet.</p> +</div> +<div class="heavy-card"> +<div class="heavy-copy"> +<p class="kicker">HEAVY QUERY ADVANTAGE</p> +<h3>The Heavier the Workload,<br/>the Clearer Paimon’s Advantage.</h3> +<p>For queries dominated by long scans, complex joins, aggregations, and sorting, Paimon’s file organization, statistics pruning, and runtime pruning translate into measurable end-to-end gains.</p> +</div> +<div class="heavy-visual"> +<div class="ring"> +<span>−12.9%</span> +<small>aggregate time</small> +</div> +<div class="heavy-times"> +<div> +<span>Paimon</span> +<strong>339.86s</strong> +</div> +<div> +<span>Parquet</span> +<strong>390.00s</strong> +</div> +</div> +</div> +</div> +<div class="wins-grid"> +<div class="win-card"> +<div class="win-top"> +<span>Q71</span> +<strong>5.4×</strong> +</div> +<div class="win-times"> +<span> +<i class="dot paimon-dot"> +</i> Paimon <!-- -->1.68s</span> +<span> +<i class="dot parquet-dot"> +</i> Parquet <!-- -->9.10s</span> +</div> +</div> +<div class="win-card"> +<div class="win-top"> +<span>Q62</span> +<strong>4.0×</strong> +</div> +<div class="win-times"> +<span> +<i class="dot paimon-dot"> +</i> Paimon <!-- -->0.62s</span> +<span> +<i class="dot parquet-dot"> +</i> Parquet <!-- -->2.50s</span> +</div> +</div> +<div class="win-card"> +<div class="win-top"> +<span>Q66</span> +<strong>3.6×</strong> +</div> +<div class="win-times"> +<span> +<i class="dot paimon-dot"> +</i> Paimon <!-- -->1.02s</span> +<span> +<i class="dot parquet-dot"> +</i> Parquet <!-- -->3.69s</span> +</div> +</div> +<div class="win-card"> +<div class="win-top"> +<span>Q57</span> +<strong>3.1×</strong> +</div> +<div class="win-times"> +<span> +<i class="dot paimon-dot"> +</i> Paimon <!-- -->3.86s</span> +<span> +<i class="dot parquet-dot"> +</i> Parquet <!-- -->11.90s</span> +</div> +</div> +</div> +</section> +<section class="meaning section-shell"> +<div class="meaning-index">02</div> +<div class="meaning-copy"> +<p class="kicker">WHY IT MATTERS</p> +<h2>This Is More Than<br/>Another Parquet Reader.</h2> +<p class="lead-paragraph">Native Parquet only needs to read files efficiently into the execution engine. Paimon must also understand snapshots, manifests, partitions, schema evolution, primary-key merges, deletion vectors, row lineage, and multiple file formats.</p> +<p>The hard part is not merely making these capabilities work, but making them interact correctly with DataFusion’s optimizer, statistics, and runtime filters. An apparently reasonable predicate pushdown can change primary-key merge results if its boundary is wrong; inaccurate null-count precision can send the optimizer toward a completely different execution plan.</p> +<p>The significance of 0.27% is not the attractive decimal. It demonstrates that <strong>a native Rust implementation can preserve full lakehouse table semantics while approaching the workload efficiency of scanning bare files.</strong> +</p> +</div> +</section> +<section class="engineering" id="engineering"> +<div class="section-shell"> +<div class="section-heading light"> +<div> +<p class="kicker">03 · ENGINEERING</p> +<h2>Performance Is Not a Switch.<br/>It Is an End-to-End Data Path.</h2> +</div> +<p>No single trick produced this result. It came from systematically aligning schemas, batching, statistics, predicates, runtime filters, and merge semantics.</p> +</div> +<div class="optimization-grid"> +<div class="optimization"> +<span>01</span> +<h3>Bring Strings Into the View Era</h3> +<p>At the DataFusion boundary, Paimon String and Binary types map to Utf8View and BinaryView. This avoids the classic Utf8 2 GiB offset limit and reduces copying and reconstruction in long-running queries.</p> +</div> +<div class="optimization"> +<span>02</span> +<h3>Carry Batch Size All the Way Down</h3> +<p><code>read.batch-size</code> no longer stops at the table-option layer. It reaches file decoding beneath raw readers, primary-key merges, and schema evolution, making throughput and memory use predictable for wide tables.</p> +</div> +<div class="optimization"> +<span>03</span> +<h3>Statistics That Are Useful and Honest</h3> +<p>Row counts, null counts, column bounds, and compressed sizes are exposed with semantic precision. When runtime filters make exactness unsafe, Paimon lowers the reported precision, giving the optimizer useful information without pretending it is exact.</p> +</div> +<div class="optimization"> +<span>04</span> +<h3>Runtime Filters Enter the Paimon Scan</h3> +<p>Equality, range, and IN conditions generated by DataFusion at runtime can be translated into Paimon predicates. Large scans wait briefly with a cost-aware upper bound, while small scans start immediately.</p> +</div> +<div class="optimization"> +<span>05</span> +<h3>Decouple Pruning From Row Filtering</h3> +<p>Predicates conservatively prune files, row groups, stripes, and pages by default. A separate switch controls whether the reader removes individual rows. The parent Filter preserves exact semantics, while the scan focuses on reading less data.</p> +</div> +<div class="optimization"> +<span>06</span> +<h3>Protect Correctness for Primary-Key and Evolved Tables</h3> +<p>Primary-key merges, data evolution, and <code>_ROW_ID</code> use semantically safe pruning boundaries. Paimon pushes down what it safely can and remains deliberately conservative where pruning could change merge results or physical positions.</p> +</div> +</div> +<div class="architecture-line" aria-label="Query-processing path from DataFusion to Paimon file reads"> +<div> +<span>01</span> +<strong>DataFusion Plan</strong> +<small>logical + physical filters</small> +</div> +<b>→</b> +<div> +<span>02</span> +<strong>Paimon Scan</strong> +<small>stats + runtime predicate</small> +</div> +<b>→</b> +<div> +<span>03</span> +<strong>Safe Pruning</strong> +<small>file / group / page</small> +</div> +<b>→</b> +<div> +<span>04</span> +<strong>Exact Result</strong> +<small>merge + parent filter</small> +</div> +</div> +</div> +</section> +<section class="ecosystem section-shell" id="ecosystem"> +<div class="section-heading"> +<div> +<p class="kicker">04 · ECOSYSTEM</p> +<h2>DataFusion Is the First Stop,<br/>Not the Destination.</h2> +</div> +<p>Rust lets Paimon Core fit naturally into embedded analytics, database backends, Python, and cross-language FFI. A high-performance, embeddable table-format core is becoming shared infrastructure for multiple engines.</p> +</div> +<div class="engine-roadmap"> +<div class="engine-card live" data-engine="DATAFUSION"> +<div class="engine-status"> +<span> +</span> Verified</div> +<h3>DataFusion</h3> +<p>From SQLContext and TableProvider to physical runtime filters, the full performance path is now connected, reaching aggregate workload parity with native Parquet at SF100.</p> +<strong>PARITY</strong> +</div> +<div class="engine-card progress" data-engine="STARROCKS"> +<div class="engine-status"> +<span> +</span> Integration in progress</div> +<h3>StarRocks</h3> +<p>Work continues on the Paimon connector, row lineage, and vector search. Paimon Rust’s native reader and FFI capabilities are opening the door to deeper integration.</p> +<strong>INTEGRATING</strong> +</div> +<div class="engine-card progress" data-engine="DORIS"> +<div class="engine-status"> +<span> +</span> Integration in progress</div> +<h3>Apache Doris</h3> +<p>The community Paimon write architecture already reserves a Rust FFI backend. Paimon Rust offers a practical path toward removing the JVM bridge and reducing data-exchange costs.</p> +<strong>INTEGRATING</strong> +</div> +</div> +<div class="ecosystem-quote"> +<span>“</span> +<p>A table-format implementation should not force every query engine to reimplement snapshot, evolution, merge, and index semantics.</p> +</div> +</section> +<section class="method" id="method"> +<div class="section-shell"> +<div class="method-heading"> +<p class="kicker">05 · BENCHMARK NOTES</p> +<h2>Honest Results Matter More<br/>Than Bigger Numbers.</h2> +</div> +<div class="method-grid"> +<dl> +<div> +<dt>Data scale</dt> +<dd>TPC-DS-derived SF100</dd> +</div> +<div> +<dt>Query set</dt> +<dd>99 queries; 96 successful on both sources</dd> +</div> +<div> +<dt>Execution engine</dt> +<dd>DataFusion 54.0.0</dd> +</div> +<div> +<dt>Paimon</dt> +<dd>paimon-rust 0.3.0 development build</dd> +</div> +</dl> +<dl> +<div> +<dt>Hardware</dt> +<dd>Apple M4 Pro · 12 cores · 48 GiB</dd> +</div> +<div> +<dt>Memory pool</dt> +<dd>32 GiB</dd> +</div> +<div> +<dt>Parallelism</dt> +<dd>12 target partitions</dd> +</div> +<div> +<dt>Run configuration</dt> +<dd>0 warmup · 1 measured iteration</dd> +</div> +</dl> +</div> +<div class="disclosure"> +<strong>Scope and limitations</strong> +<p>This is a TPC-DS-derived, non-TPC benchmark, not an official TPC result. Both sources hit the same 32 GiB resource limit on Q67, Q78, and Q97, so the aggregate comparison includes only the 96 queries that succeeded on both. Importing into Paimon rewrites physical files. A single measured run is suitable for demonstrating the current engineering milestone, but it does not replace a formal performance study with multiple runs, separated cold and warm caches, object storage, and multi-node environments.</p> +</div> +</div> +</section> +<section class="next section-shell"> +<p class="kicker">WHAT'S NEXT</p> +<h2>SF100 Parity Is Here.<br/>Next, We Take It Toward Production.</h2> +<div class="next-grid"> +<div> +<span>01</span> +<h3>SF1000</h3> +<p>Validate larger datasets, more spilling, and longer scan pipelines.</p> +</div> +<div> +<span>02</span> +<h3>Cloud Storage</h3> +<p>Quantify range-read, caching, and concurrency gains on S3, OSS, and other object stores.</p> +</div> +<div> +<span>03</span> +<h3>Multi-engine</h3> +<p>Let Doris, StarRocks, and more engines share the same Rust table-format capabilities.</p> +</div> +<div> +<span>04</span> +<h3>Short Queries</h3> +<p>Keep reducing fixed costs in manifests, planning, and runtime-filter coordination.</p> +</div> +</div> +<div class="cta-card"> +<div> +<span>BUILD THE LAKEHOUSE CORE ONCE.</span> +<h3>Let Every Engine Understand Paimon Natively.</h3> +</div> +<a class="button primary" href="https://github.com/apache/paimon-rust"; target="_blank" rel="noreferrer">Join Paimon Rust ↗</a> +</div> +</section> +<footer> +<div class="section-shell footer-inner"> +<div> +<div class="brand footer-brand"> +<span class="brand-mark">P</span> +<span>PAIMON RUST</span> +</div> +<p>A high-performance lakehouse table-format core for multiple engines.</p> +</div> +<div class="references"> +<a href="https://github.com/apache/paimon-rust"; target="_blank" rel="noreferrer">Paimon Rust Project<!-- --> ↗</a> +<a href="https://github.com/apache/paimon-rust/pull/549"; target="_blank" rel="noreferrer">Runtime Filters and End-to-End Performance #549<!-- --> ↗</a> +<a href="https://github.com/apache/doris/pull/65381"; target="_blank" rel="noreferrer">Doris Paimon Write Architecture and Rust FFI Hook #65381<!-- --> ↗</a> +<a href="https://github.com/StarRocks/starrocks/pull/73048"; target="_blank" rel="noreferrer">StarRocks Paimon Vector Search #73048<!-- --> ↗</a> +</div> +<small>Community engineering update · July 20, 2026 · Results from a locally reproducible experiment</small> Review Comment: Blocking: the page labels this a locally reproducible experiment, but the PR contains neither raw benchmark JSON nor enough metadata to reproduce or verify the numbers. The benchmark README explicitly requires the exact paimon-rust commit, generated-data scale and generator version, file counts and physical bytes, OS/CPU/memory/storage model, and exact commands for every published report. Please add those details and link or commit the two raw result files so reviewers can verify the totals, failed queries, and the 30/96 count. ########## docs/src/html/benchmark/index.html: ########## @@ -0,0 +1,434 @@ +<!DOCTYPE html> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> +<html lang="en"> +<head> +<meta charSet="utf-8"/> +<meta name="viewport" content="width=device-width, initial-scale=1"/> +<link rel="stylesheet" href="styles.css"/> +<title>Paimon Rust Reaches a Performance Turning Point | TPC-DS SF100</title> +<meta name="description" content="On DataFusion 54, Paimon Rust finishes the comparable TPC-DS SF100 queries within 0.27% of native Parquet, while the heavy-query group is 12.9% faster."/> +<meta property="og:title" content="Paimon Rust Reaches a Performance Turning Point"/> +<meta property="og:description" content="TPC-DS SF100: total workload execution time is within 0.27% of DataFusion native Parquet."/> +<meta property="og:type" content="article"/> +<meta property="og:locale" content="en_US"/> +<meta name="twitter:card" content="summary"/> +<meta name="twitter:title" content="Paimon Rust Reaches a Performance Turning Point"/> +<meta name="twitter:description" content="TPC-DS SF100: total workload execution time is within 0.27% of DataFusion native Parquet."/> +</head> +<body> +<main> +<script type="application/ld+json">{"@context":"https://schema.org","@type":"TechArticle","headline":"Paimon Rust Reaches a Performance Turning Point: TPC-DS SF100 Parity with Native Parquet on DataFusion","datePublished":"2026-07-20","inLanguage":"en","author":{"@type":"Organization","name":"Paimon Rust Community"},"description":"Paimon Rust finishes the comparable TPC-DS SF100 queries within 0.27% of DataFusion native Parquet."}</script> +<header class="topbar"> +<a class="brand" href="#top" aria-label="Back to the top"> +<span class="brand-mark">P</span> +<span>PAIMON RUST</span> +</a> +<nav aria-label="Article navigation"> +<a href="#result">Results</a> +<a href="#engineering">Engineering</a> +<a href="#ecosystem">Ecosystem</a> +<a href="#method">Methodology</a> +</nav> +<a class="github-link" href="https://github.com/apache/paimon-rust"; target="_blank" rel="noreferrer">GitHub ↗</a> +</header> +<article id="top"> +<section class="hero section-shell"> +<div class="hero-glow" aria-hidden="true"> +</div> +<div class="eyebrow"> +<span> +</span> PERFORMANCE MILESTONE · JUL 2026</div> +<h1><strong>Paimon Rust</strong> <em>At Native Speed</em> +</h1> +<p class="hero-lead">In a TPC-DS SF100 workload on DataFusion 54, Paimon Rust finishes the 96 comparable queries <strong>within 0.27% of native Parquet</strong>. For the queries that dominate execution time, Paimon is 12.9% faster in aggregate. Table semantics, snapshots, schema evolution, and lakehouse capabilities no longer need to come with a material analytical-performance penalty.</p> +<div class="hero-actions"> +<a class="button primary" href="#result">View Results</a> +<a class="button secondary" href="https://github.com/apache/paimon-rust/pull/549"; target="_blank" rel="noreferrer">Read the Implementation PR ↗</a> +</div> +<div class="metric-strip" role="list" aria-label="Key performance results"> +<div class="metric" role="listitem"> +<span class="metric-value">+0.27%</span> +<span class="metric-label">Total workload time gap</span> +</div> +<div class="metric" role="listitem"> +<span class="metric-value">30 / 96</span> +<span class="metric-label">Queries won by Paimon</span> +</div> +<div class="metric" role="listitem"> +<span class="metric-value accent">−12.9%</span> +<span class="metric-label">Aggregate heavy-query time</span> +</div> +<div class="metric" role="listitem"> +<span class="metric-value">5.4×</span> +<span class="metric-label">Q71 speedup</span> +</div> +</div> +</section> +<section class="statement section-shell"> +<p class="kicker">THE SHORT VERSION</p> +<blockquote>We are no longer asking whether Paimon in Rust can run.<br/>The question now is: <mark>how far can it go?</mark> +</blockquote> +</section> +<section class="results section-shell" id="result"> +<div class="section-heading"> +<div> +<p class="kicker">01 · THE RESULT</p> +<h2>The Same DataFusion.<br/>Paimon Now Stands Beside Parquet.</h2> +</div> +<p>This benchmark uses the same DataFusion SQL, planner, and execution operators to compare a Paimon source with DataFusion native Parquet. Because importing into Paimon rewrites the physical files, this is an end-to-end source comparison rather than a measurement of catalog overhead alone.</p> +</div> +<div class="score-card"> +<div class="score-head"> +<div> +<span class="score-label">96 queries successful on both sources · total execution time</span> +<strong>1.39 seconds</strong> +<small>Absolute difference across the entire workload</small> +</div> +<span class="parity-pill">PARITY REACHED</span> +</div> +<div class="bar-chart" aria-label="Total execution time: Paimon 516.44 seconds, Parquet 515.05 seconds"> +<div class="bar-row"> +<div class="bar-meta"> +<span>Paimon Rust</span> +<strong>516.44s</strong> +</div> +<div class="bar-track"> +<div class="bar paimon" style="width:100%"> +</div> +</div> +</div> +<div class="bar-row"> +<div class="bar-meta"> +<span>Native Parquet</span> +<strong>515.05s</strong> +</div> +<div class="bar-track"> +<div class="bar parquet" style="width:99.73%"> +</div> +</div> +</div> +</div> +<p class="score-note">“Parity” describes the aggregate workload, not identical performance on every query. Paimon still pays fixed overhead on some short queries. Among the 23 heavy queries for which Parquet takes at least five seconds, however, Paimon finishes in 339.86 seconds versus 390.00 seconds for native Parquet.</p> +</div> +<div class="heavy-card"> +<div class="heavy-copy"> +<p class="kicker">HEAVY QUERY ADVANTAGE</p> +<h3>The Heavier the Workload,<br/>the Clearer Paimon’s Advantage.</h3> +<p>For queries dominated by long scans, complex joins, aggregations, and sorting, Paimon’s file organization, statistics pruning, and runtime pruning translate into measurable end-to-end gains.</p> +</div> +<div class="heavy-visual"> +<div class="ring"> +<span>−12.9%</span> +<small>aggregate time</small> +</div> +<div class="heavy-times"> +<div> +<span>Paimon</span> +<strong>339.86s</strong> +</div> +<div> +<span>Parquet</span> +<strong>390.00s</strong> +</div> +</div> +</div> +</div> +<div class="wins-grid"> +<div class="win-card"> +<div class="win-top"> +<span>Q71</span> +<strong>5.4×</strong> +</div> +<div class="win-times"> +<span> +<i class="dot paimon-dot"> +</i> Paimon <!-- -->1.68s</span> +<span> +<i class="dot parquet-dot"> +</i> Parquet <!-- -->9.10s</span> +</div> +</div> +<div class="win-card"> +<div class="win-top"> +<span>Q62</span> +<strong>4.0×</strong> +</div> +<div class="win-times"> +<span> +<i class="dot paimon-dot"> +</i> Paimon <!-- -->0.62s</span> +<span> +<i class="dot parquet-dot"> +</i> Parquet <!-- -->2.50s</span> +</div> +</div> +<div class="win-card"> +<div class="win-top"> +<span>Q66</span> +<strong>3.6×</strong> +</div> +<div class="win-times"> +<span> +<i class="dot paimon-dot"> +</i> Paimon <!-- -->1.02s</span> +<span> +<i class="dot parquet-dot"> +</i> Parquet <!-- -->3.69s</span> +</div> +</div> +<div class="win-card"> +<div class="win-top"> +<span>Q57</span> +<strong>3.1×</strong> +</div> +<div class="win-times"> +<span> +<i class="dot paimon-dot"> +</i> Paimon <!-- -->3.86s</span> +<span> +<i class="dot parquet-dot"> +</i> Parquet <!-- -->11.90s</span> +</div> +</div> +</div> +</section> +<section class="meaning section-shell"> +<div class="meaning-index">02</div> +<div class="meaning-copy"> +<p class="kicker">WHY IT MATTERS</p> +<h2>This Is More Than<br/>Another Parquet Reader.</h2> +<p class="lead-paragraph">Native Parquet only needs to read files efficiently into the execution engine. Paimon must also understand snapshots, manifests, partitions, schema evolution, primary-key merges, deletion vectors, row lineage, and multiple file formats.</p> +<p>The hard part is not merely making these capabilities work, but making them interact correctly with DataFusion’s optimizer, statistics, and runtime filters. An apparently reasonable predicate pushdown can change primary-key merge results if its boundary is wrong; inaccurate null-count precision can send the optimizer toward a completely different execution plan.</p> +<p>The significance of 0.27% is not the attractive decimal. It demonstrates that <strong>a native Rust implementation can preserve full lakehouse table semantics while approaching the workload efficiency of scanning bare files.</strong> +</p> +</div> +</section> +<section class="engineering" id="engineering"> +<div class="section-shell"> +<div class="section-heading light"> +<div> +<p class="kicker">03 · ENGINEERING</p> +<h2>Performance Is Not a Switch.<br/>It Is an End-to-End Data Path.</h2> +</div> +<p>No single trick produced this result. It came from systematically aligning schemas, batching, statistics, predicates, runtime filters, and merge semantics.</p> +</div> +<div class="optimization-grid"> +<div class="optimization"> +<span>01</span> +<h3>Bring Strings Into the View Era</h3> +<p>At the DataFusion boundary, Paimon String and Binary types map to Utf8View and BinaryView. This avoids the classic Utf8 2 GiB offset limit and reduces copying and reconstruction in long-running queries.</p> +</div> +<div class="optimization"> +<span>02</span> +<h3>Carry Batch Size All the Way Down</h3> +<p><code>read.batch-size</code> no longer stops at the table-option layer. It reaches file decoding beneath raw readers, primary-key merges, and schema evolution, making throughput and memory use predictable for wide tables.</p> +</div> +<div class="optimization"> +<span>03</span> +<h3>Statistics That Are Useful and Honest</h3> +<p>Row counts, null counts, column bounds, and compressed sizes are exposed with semantic precision. When runtime filters make exactness unsafe, Paimon lowers the reported precision, giving the optimizer useful information without pretending it is exact.</p> +</div> +<div class="optimization"> +<span>04</span> +<h3>Runtime Filters Enter the Paimon Scan</h3> +<p>Equality, range, and IN conditions generated by DataFusion at runtime can be translated into Paimon predicates. Large scans wait briefly with a cost-aware upper bound, while small scans start immediately.</p> +</div> +<div class="optimization"> +<span>05</span> +<h3>Decouple Pruning From Row Filtering</h3> +<p>Predicates conservatively prune files, row groups, stripes, and pages by default. A separate switch controls whether the reader removes individual rows. The parent Filter preserves exact semantics, while the scan focuses on reading less data.</p> +</div> +<div class="optimization"> +<span>06</span> +<h3>Protect Correctness for Primary-Key and Evolved Tables</h3> +<p>Primary-key merges, data evolution, and <code>_ROW_ID</code> use semantically safe pruning boundaries. Paimon pushes down what it safely can and remains deliberately conservative where pruning could change merge results or physical positions.</p> +</div> +</div> +<div class="architecture-line" aria-label="Query-processing path from DataFusion to Paimon file reads"> +<div> +<span>01</span> +<strong>DataFusion Plan</strong> +<small>logical + physical filters</small> +</div> +<b>→</b> +<div> +<span>02</span> +<strong>Paimon Scan</strong> +<small>stats + runtime predicate</small> +</div> +<b>→</b> +<div> +<span>03</span> +<strong>Safe Pruning</strong> +<small>file / group / page</small> +</div> +<b>→</b> +<div> +<span>04</span> +<strong>Exact Result</strong> +<small>merge + parent filter</small> +</div> +</div> +</div> +</section> +<section class="ecosystem section-shell" id="ecosystem"> +<div class="section-heading"> +<div> +<p class="kicker">04 · ECOSYSTEM</p> +<h2>DataFusion Is the First Stop,<br/>Not the Destination.</h2> +</div> +<p>Rust lets Paimon Core fit naturally into embedded analytics, database backends, Python, and cross-language FFI. A high-performance, embeddable table-format core is becoming shared infrastructure for multiple engines.</p> +</div> +<div class="engine-roadmap"> +<div class="engine-card live" data-engine="DATAFUSION"> +<div class="engine-status"> +<span> +</span> Verified</div> +<h3>DataFusion</h3> +<p>From SQLContext and TableProvider to physical runtime filters, the full performance path is now connected, reaching aggregate workload parity with native Parquet at SF100.</p> +<strong>PARITY</strong> +</div> +<div class="engine-card progress" data-engine="STARROCKS"> +<div class="engine-status"> +<span> +</span> Integration in progress</div> +<h3>StarRocks</h3> +<p>Work continues on the Paimon connector, row lineage, and vector search. Paimon Rust’s native reader and FFI capabilities are opening the door to deeper integration.</p> +<strong>INTEGRATING</strong> +</div> +<div class="engine-card progress" data-engine="DORIS"> +<div class="engine-status"> +<span> +</span> Integration in progress</div> +<h3>Apache Doris</h3> +<p>The community Paimon write architecture already reserves a Rust FFI backend. Paimon Rust offers a practical path toward removing the JVM bridge and reducing data-exchange costs.</p> +<strong>INTEGRATING</strong> +</div> +</div> +<div class="ecosystem-quote"> +<span>“</span> +<p>A table-format implementation should not force every query engine to reimplement snapshot, evolution, merge, and index semantics.</p> +</div> +</section> +<section class="method" id="method"> +<div class="section-shell"> +<div class="method-heading"> +<p class="kicker">05 · BENCHMARK NOTES</p> +<h2>Honest Results Matter More<br/>Than Bigger Numbers.</h2> +</div> +<div class="method-grid"> +<dl> +<div> +<dt>Data scale</dt> +<dd>TPC-DS-derived SF100</dd> +</div> +<div> +<dt>Query set</dt> +<dd>99 queries; 96 successful on both sources</dd> +</div> +<div> +<dt>Execution engine</dt> +<dd>DataFusion 54.0.0</dd> +</div> +<div> +<dt>Paimon</dt> +<dd>paimon-rust 0.3.0 development build</dd> +</div> +</dl> +<dl> +<div> +<dt>Hardware</dt> +<dd>Apple M4 Pro · 12 cores · 48 GiB</dd> +</div> +<div> +<dt>Memory pool</dt> +<dd>32 GiB</dd> +</div> +<div> +<dt>Parallelism</dt> +<dd>12 target partitions</dd> +</div> +<div> +<dt>Run configuration</dt> +<dd>0 warmup · 1 measured iteration</dd> Review Comment: Blocking: this result uses 0 warmups and a single measured iteration, while `benchmarks/tpcds/README.md`'s published procedure uses `--warmup 1 --iterations 3`. The headline gap is only 1.39 seconds over roughly 515 seconds (0.27%), which can easily be smaller than run-to-run or cache noise; one observation cannot establish parity. Please rerun both sources for multiple measured iterations under the same cache policy (ideally alternating source order), publish an aggregate such as the median plus dispersion, and derive the headline numbers from that dataset. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
