This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 855af91 Commit build products
855af91 is described below
commit 855af9154aea006f3700752f4e190e4c4c00fdb0
Author: Build Pelican (action) <[email protected]>
AuthorDate: Wed Mar 26 13:13:39 2025 +0000
Commit build products
---
output/2025/03/24/datafusion-46.0.0/index.html | 147 +++++++++++++++++++++
...anci-and-berkay-sahin-on-behalf-of-the-pmc.html | 104 +++++++++++++++
output/category/blog.html | 35 +++++
output/feed.xml | 18 ++-
output/feeds/all-en.atom.xml | 104 ++++++++++++++-
output/feeds/blog.atom.xml | 104 ++++++++++++++-
...-and-berkay-sahin-on-behalf-of-the-pmc.atom.xml | 104 +++++++++++++++
...i-and-berkay-sahin-on-behalf-of-the-pmc.rss.xml | 18 +++
.../datafusion-46.0.0/diagnostic-example.png | Bin 0 -> 140405 bytes
output/index.html | 35 +++++
10 files changed, 666 insertions(+), 3 deletions(-)
diff --git a/output/2025/03/24/datafusion-46.0.0/index.html
b/output/2025/03/24/datafusion-46.0.0/index.html
new file mode 100644
index 0000000..8e11f06
--- /dev/null
+++ b/output/2025/03/24/datafusion-46.0.0/index.html
@@ -0,0 +1,147 @@
+<!doctype html>
+<html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>Apache DataFusion 46.0.0 Released - Apache DataFusion Blog</title>
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth
navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog"><img
src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache
DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse"
data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false"
aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+
+<!-- page contents -->
+<div id="contents">
+ <div class="bg-white p-5 rounded">
+ <div class="col-sm-8 mx-auto">
+ <h1>
+ Apache DataFusion 46.0.0 Released
+ </h1>
+ <p>Posted on: Mon 24 March 2025 by Oznur Hanci and Berkay Sahin
on behalf of the PMC</p>
+ <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>We’re excited to announce the release of <strong>Apache
DataFusion 46.0.0</strong>! This new version represents a significant milestone
for the project, packing in a wide range of improvements and fixes. You can
find the complete details in the full <a
href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md";>changelog</a>.
We’ll highlight the most important changes below and guide you through
upgrading.</p>
+<h2>Breaking Changes</h2>
+<p>DataFusion 46.0.0 brings a few <strong>breaking
changes</strong> that may require adjustments to your code as described in
the <a
href="https://datafusion.apache.org/library-user-guide/upgrading.html";>Upgrade
Guide</a>. Here are the most notable ones:</p>
+<ul>
+<li><a href="https://github.com/apache/datafusion/pull/14224#";>Unified
<code>DataSourceExec</code> Execution
Plan</a><strong>:</strong> DataFusion 46.0.0 introduces a major refactor
of scan operators. The separate file-format-specific execution plan nodes
(<code>ParquetExec</code>, <code>CsvExec</code>, <code>JsonExec</code>, <code>AvroExec</code>,
etc.) have been <strong>deprecated and merged into a single
<code>DataSourceExec</code> plan</strong>. Format-s [...]
+<li><a
href="https://github.com/apache/arrow-datafusion/issues/7360#:~:text=2";>**Error
Handling Improvements</a> (<code>DataFusionError::Collection</code>):** We
began overhauling DataFusion’s approach to error handling. In this
release, a new error
variant <code>DataFusionError::Collection</code> (and related
mechanisms) has been introduced to aggregate multiple errors into one. This is
part of a broader effort to provide richer error context and reduce internal
pan [...]
+</ul>
+<h2>Performance Improvements</h2>
+<p>DataFusion 46.0.0 comes with a slew of performance enhancements across the
board. Here are some of the noteworthy optimizations in this release:</p>
+<ul>
+<li><strong>Faster <code>median()</code> (no
grouping):</strong> The <code>median()</code> aggregate function
got a special fast path when used without a <code>GROUP BY</code>. By
optimizing its accumulator, median calculation is about <strong>2×
faster</strong> in the single-group case. If you
use <code>MEDIAN()</code> on large datasets (especially as a single
value), you should notice reduced query times (PR <a href="https://githu [...]
+<li><strong>Optimized <code>FIRST_VALUE</code>/<code>LAST_VALUE</code>:</strong> The <code>FIRST_VALUE</code> and <code>LAST_VALUE</code> window
functions have been improved by avoiding an internal sort of rows. Instead of
sorting each partition, the implementation now uses a direct approach to pick
the first/last element. This yields <strong>10–100% performance
improvement</strong> for these functions, depending on the scenario.
Queries usin [...]
+<li><strong><code>repeat()</code> String Function
Boost:</strong> Repeating strings is now more efficient –
the <code>repeat(text, n)</code> function was optimized by
about <strong>50%</strong>. This was achieved by reducing allocations and
using a more efficient concatenation strategy. If you generate large repeated
strings in queries, this can cut the time nearly in half (PR <a
href="https://github.com/apache/datafusion/pull/14697";>#14697</a> by <a href="
[...]
+<li><strong>Ultra-fast <code>uuid()</code> UDF:</strong> The <code>uuid()</code> function
(which generates random UUID strings) received a major speed-up. It’s
now roughly <strong>40× faster</strong> than before! The new
implementation avoids unnecessary string copying and uses a more direct
conversion to hex, making bulk UUID generation far more practical (PR <a
href="https://github.com/apache/datafusion/pull/14675";>#14675</a> by <a href="h
[...]
+<li><strong>Accelerated <code>chr()</code> and <code>to_hex()</code>:</strong> Several
scalar functions have been micro-optimized.
The <code>chr()</code> function (which returns the character for a
given ASCII code) is about <strong>4× faster</strong> now, and
the <code>to_hex()</code> function (which converts numbers to hex
string) is roughly <strong>2× faster</strong>. These improvements
may be most noticeable in tight [...]
+<li><strong>No More RowConverter in Grouped Ordering:</strong> We removed
an inefficient step in the <em>partial grouping</em> algorithm.
The <code>GroupOrderingPartial</code> operator no longer converts
data to “row format” for each batch
(via <code>RowConverter</code>). Instead, it uses a direct arrow-based
approach to detect sort key changes. This eliminated overhead and yields a nice
speedup for certain aggregation queries. (PR <a href="https [...]
+<li><strong>Predicate Pruning for <code>NOT
LIKE</code>:</strong> DataFusion’s parquet reader can now prune row
groups using <code>NOT LIKE</code> filters, similar to how it
handles <code>LIKE</code>. This means if you have a filter such
as <code>column NOT LIKE 'prefix%'</code>, DataFusion can use min/max
statistics to skip reading files/parts that can be determined to either
entirely match or not match the predicate. In particular, a pattern like&nbs
[...]
+</ul>
+<h2>Google Summer of Code 2025</h2>
+<p>Another exciting development: <strong>Apache DataFusion has been
accepted as a mentoring organization for Google Summer of Code (GSoC)
2025</strong>! 🎉 This means that this summer, students from around the world
will have the opportunity to contribute to DataFusion under the guidance of our
committers. We have put together <a
href="https://datafusion.apache.org/contributor-guide/gsoc_project_ideas.html";>a
list of project ideas</a> that candidates can choose from.</p>
+<p>If you’re interested, check out our <a
href="https://datafusion.apache.org/contributor-guide/gsoc_application_guidelines.html";>GSoC
Application Guidelines</a>. We encourage students to reach out, discuss ideas
with us, and apply.</p>
+<h2>Highlighted New Features</h2>
+<h3>Improved Diagnostics</h3>
+<p>DataFusion 46.0.0 introduces a new <a
href="https://github.com/apache/datafusion/issues/14429";><strong>SQL
Diagnostics framework</strong></a> to make error messages more
understandable. This comes in the form of
new <code>Diagnostic</code> and <code>DiagnosticEntry</code> types,
which allow the system to attach rich context (like source query text spans)
to error messages. In practical terms, certain planner errors will now point to
the exact location in [...]
+<p>For example, if you reference an unknown table or miss a column in
<code>GROUP BY</code> the error message will include the query snippet causing
the error. These diagnostics are meant for end-users of applications built on
DataFusion, providing clearer messages instead of generic errors. Here’s
an example:</p>
+<p><img alt="diagnostic-example" class="img-responsive"
src="/blog/images/datafusion-46.0.0/diagnostic-example.png" width="80%"/></p>
+<p>Currently, diagnostics cover unresolved table/column references, missing
<code>GROUP BY</code> columns, ambiguous references, wrong number of UNION
columns, type mismatches, and a few others. Future releases will extend this to
more error types. This feature should greatly ease debugging of complex SQL by
pinpointing errors directly in the query text. We thank <a
href="https://github.com/eliaperantoni";>@eliaperantoni</a> for his
contributions in this project.</p>
+<h3>Unified <code>DataSourceExec</code> for Table Providers</h3>
+<p>As mentioned, DataFusion now uses a
unified <code>DataSourceExec</code> for reading tables, which is both
a breaking change and a feature. <em>Why is this important?</em> The
new approach simplifies how custom table providers are integrated and
optimized. Namely, the optimizer can treat file scans uniformly and push down
filters/limits more consistently when there is one execution plan that handles
all data sources. The new <code>DataSourceExec</code> is [...]
+<p>All built-in sources (Parquet, CSV, Avro, Arrow, JSON, etc.) have been
migrated to this framework. This unification makes the codebase cleaner and
sets the stage for future enhancements (like consistent metadata handling and
limit pushdown across all formats). Check out PR <a
href="https://github.com/apache/datafusion/pull/14224";>#14224</a> for design
details. We thank <a
href="https://github.com/mertak-synnada";>@mertak-synnada</a> and <a
href="https://github.com/ozankabak";>@ozankabak [...]
+<h3>FFI Support for Scalar UDFs</h3>
+<p>DataFusion’s Foreign Function Interface (FFI) has been extended to
support <a
href="https://github.com/apache/datafusion/pull/14579";><strong>user-defined
scalar functions</strong></a> defined in external languages. In 46.0.0,
you can now expose a custom scalar UDF through the FFI layer and use it in
DataFusion as if it were built-in. This is particularly exciting for the
<strong>Python bindings</strong> and other language integrations – it
means you could define [...]
+<h3>New Statistics/Distribution Framework</h3>
+<p>This release, thanks mainly to <a
href="https://github.com/Fly-Style";>@Fly-Style</a> with contributions from <a
href="https://github.com/ozankabak";>@ozankabak</a> and <a
href="https://github.com/berkaysynnada";>@berkaysynnada</a>, includes the
initial pieces of a <a
href="https://github.com/apache/datafusion/pull/14699";>**redesigned statistics
framework</a>.<strong> DataFusion’s optimizer can now represent column
data distributions using a new <code>Distribution</code>& [...]
+<p>For example, if a filter expression is applied to a column with a known
uniform distribution range, the optimizer can propagate that to estimate result
selectivity more accurately. Similarly, comparisons (<code>=</code>,
<code>></code>, etc.) on columns yield Bernoulli distributions (with
true/false probabilities) in this model.</p>
+<p>This is a foundational change with many follow-on PRs underway. Even though
the immediate user-visible effect is limited (the optimizer didn't magically
improve by an order of magnitude overnight), but it lays groundwork for more
advanced query planning in the future. Over time, as statistics information
encapsulated in <code>Distribution</code>s get integrated, DataFusion will be
able to make smarter decisions like more aggressive parquet pruning, better
join orderings, and so on bas [...]
+<h3>Aggregate Monotonicity and Window Ordering</h3>
+<p>DataFusion 46.0.0 adds a new concept of <a
href="https://github.com/apache/datafusion/pull/14271";><strong>set-monotonicity</strong></a>
for certain transformations, which helps avoid unnecessary sort operations. In
particular, the planner now understands when a <strong>window function
introduces new orderings of data</strong>.</p>
+<p>For example, DataFusion now recognizes that a window-aggregate like
<code>MAX</code> on a column can produce a result that is <strong>monotonically
increasing</strong>, even if the input column is unordered — depending on
the window frame used.</p>
+<p>Consider the following query:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span>
<span class="k">MAX</span><span class="p">(</span><span
class="n">c1</span><span class="p">)</span> <span class="n">OVER</span> <span
class="p">(</span>
+ <span class="k">ROWS</span> <span class="k">BETWEEN</span> <span
class="n">UNBOUNDED</span> <span class="n">PRECEDING</span> <span
class="k">AND</span> <span class="k">CURRENT</span> <span class="k">ROW</span>
+<span class="p">)</span> <span class="k">AS</span> <span
class="n">max_c1</span>
+<span class="k">FROM</span> <span class="n">c1_table</span>
+<span class="k">ORDER</span> <span class="k">BY</span> <span
class="n">max_c1</span><span class="p">;</span>
+</code></pre></div>
+<p>In earlier versions of DataFusion, this query would require an additional
SortExec on max_c1 to satisfy the ORDER BY clause. However, with the new
set-monotonicity logic, the planner knows that MAX(...) OVER (...) produces
values that are not smaller than the previous row, making the extra sort
redundant. This leads to more efficient query execution.</p>
+<p>PR <a href="https://github.com/apache/datafusion/pull/14271";>#14271</a>
introduced the core monotonicity tracking for aggregates and window functions.
+PR <a href="https://github.com/apache/datafusion/pull/14813";>#14813</a>
improved ordering preservation within various window frame types, and brought
an extensive test coverage.
+Huge thanks to <a href="https://github.com/berkaysynnada";>@berkaysynnada</a>
and <a href="https://github.com/mertak-synnada";>@mertak-synnada</a> for
designing and implementing this optimizer enhancement!</p>
+<h3>UNION [ALL | DISTINCT] BY NAME Support</h3>
+<p>DataFusion now supports UNION BY NAME and UNION ALL BY NAME, which align
columns by name instead of position. This matches functionality found in
systems like Spark and DuckDB and simplifies combining heterogeneously ordered
result sets.</p>
+<p>You no longer need to rewrite column order manually — just write:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span>
<span class="n">col1</span><span class="p">,</span> <span class="n">col2</span>
<span class="k">FROM</span> <span class="n">t1</span>
+<span class="k">UNION</span> <span class="k">ALL</span> <span
class="k">BY</span> <span class="n">NAME</span>
+<span class="k">SELECT</span> <span class="n">col2</span><span
class="p">,</span> <span class="n">col1</span> <span class="k">FROM</span>
<span class="n">t2</span><span class="p">;</span>
+</code></pre></div>
+<p>Under the hood, this is supported by the new union_by_name() and
union_by_name_distinct() plan builder methods.</p>
+<p>Thanks to <a href="https://github.com/rkrishn7";>@rkrishn7</a> for PR <a
href="https://github.com/apache/datafusion/pull/14538";>#14538</a>.</p>
+<h3>New range() Table Function</h3>
+<p>A new table-valued function range(start, stop, step) has been added to make
it easy to generate integer sequences — similar to PostgreSQL’s
generate_series() or Spark’s range().</p>
+<p>Example:</p>
+<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span>
<span class="o">*</span> <span class="k">FROM</span> <span
class="n">range</span><span class="p">(</span><span class="mi">1</span><span
class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span
class="mi">2</span><span class="p">);</span>
+</code></pre></div>
+<p>This returns: 1, 3, 5, 7, 9. It’s great for testing, cross joins,
surrogate keys, and more.</p>
+<p>Thanks to <a href="https://github.com/simonvandel";>@simonvandel</a> for PR
<a href="https://github.com/apache/datafusion/pull/14830";>#14830</a>.</p>
+<h2>Upgrade Guide and Changelog</h2>
+<p>Upgrading to 46.0.0 should be straightforward for most users, but do review
the <a
href="https://datafusion.apache.org/library-user-guide/upgrading.html";>Upgrade
Guide for DataFusion 46.0.0</a> for detailed steps and code changes. The
upgrade guide covers the breaking changes mentioned (like replacing old exec
nodes with <code>DataSourceExec</code>, updating UDF invocation
to <code>invoke_with_args</code>, etc.) and provides code snippets to help
with the transitio [...]
+<h2>Get Involved</h2>
+<p>Apache DataFusion is an open-source project, and we welcome involvement
from anyone interested. Now is a great time to take 46.0.0 for a spin: try it
out on your workloads, and let us know if you encounter any issues or have
suggestions. You can report bugs or request features on our GitHub issue
tracker, or better yet, submit a pull request. Join our community discussions
– whether you have questions, want to share how you’re using
DataFusion, or are looking to contr [...]
+<p>Happy querying!</p>
+ </div>
+ </div>
+ </div>
+ <!-- footer -->
+ <div class="row">
+ <div class="large-12 medium-12 columns">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2025, <a href="https://www.apache.org/";>The Apache
Software Foundation</a>, Licensed under the <a
href="https://www.apache.org/licenses/LICENSE-2.0";>Apache License, Version
2.0</a>.<br/>
+ Apache® and the Apache feather logo are trademarks of The Apache
Software Foundation.
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
+ </body>
+</html>
diff --git
a/output/author/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.html
b/output/author/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.html
new file mode 100644
index 0000000..c6c7e3a
--- /dev/null
+++ b/output/author/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.html
@@ -0,0 +1,104 @@
+ <!doctype html>
+ <html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>Apache DataFusion Blog</title>
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> <link
href="/blog/css/blog_index.css" rel="stylesheet">
+ </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+ <div>
+
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth
navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog"><img
src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache
DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse"
data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false"
aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ </div>
+ </div>
+</nav>
+ <div id="contents">
+ <div class="bg-white p-5 rounded">
+ <div class="col-sm-8 mx-auto">
+<div id="contents">
+ <div class="bg-white p-5 rounded">
+ <div class="col-sm-8 mx-auto">
+
+ <h3>Welcome to the Apache DataFusion Blog!</h3>
+ <p><i>Here you can find the latest updates from DataFusion and
related projects.</i></p>
+
+
+ <!-- Post -->
+ <div class="row">
+ <div class="callout">
+ <article class="post">
+ <header>
+ <div class="title">
+ <h1><a
href="/blog/2025/03/24/datafusion-46.0.0">Apache DataFusion 46.0.0
Released</a></h1>
+ <p>Posted on: Mon 24 March 2025 by Oznur Hanci and
Berkay Sahin on behalf of the PMC</p>
+ <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>We’re excited to announce the release of <strong>Apache
DataFusion 46.0.0</strong>! This new version represents a significant milestone
for the project, packing in a wide range of improvements and fixes. You can
find the complete details in the full <a
href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md";>changelog</a>.
We’ll highlight the most important changes below …</p></p>
+ <footer>
+ <ul class="actions">
+ <div style="text-align: right"><a
href="/blog/2025/03/24/datafusion-46.0.0" class="button medium">Continue
Reading</a></div>
+ </ul>
+ <ul class="stats">
+ </ul>
+ </footer>
+ </article>
+ </div>
+ </div>
+
+ </div>
+ </div>
+</div> </div>
+ </div>
+ </div>
+
+ <!-- footer -->
+ <div class="row">
+ <div class="large-12 medium-12 columns">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2025, <a href="https://www.apache.org/";>The Apache
Software Foundation</a>, Licensed under the <a
href="https://www.apache.org/licenses/LICENSE-2.0";>Apache License, Version
2.0</a>.<br/>
+ Apache® and the Apache feather logo are trademarks of The Apache
Software Foundation.
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </div>
+ </main>
+ </body>
+ </html>
diff --git a/output/category/blog.html b/output/category/blog.html
index 33393cc..e59878e 100644
--- a/output/category/blog.html
+++ b/output/category/blog.html
@@ -47,6 +47,41 @@
<p><i>Here you can find the latest updates from DataFusion and
related projects.</i></p>
+ <!-- Post -->
+ <div class="row">
+ <div class="callout">
+ <article class="post">
+ <header>
+ <div class="title">
+ <h1><a
href="/blog/2025/03/24/datafusion-46.0.0">Apache DataFusion 46.0.0
Released</a></h1>
+ <p>Posted on: Mon 24 March 2025 by Oznur Hanci and
Berkay Sahin on behalf of the PMC</p>
+ <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>We’re excited to announce the release of <strong>Apache
DataFusion 46.0.0</strong>! This new version represents a significant milestone
for the project, packing in a wide range of improvements and fixes. You can
find the complete details in the full <a
href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md";>changelog</a>.
We’ll highlight the most important changes below …</p></p>
+ <footer>
+ <ul class="actions">
+ <div style="text-align: right"><a
href="/blog/2025/03/24/datafusion-46.0.0" class="button medium">Continue
Reading</a></div>
+ </ul>
+ <ul class="stats">
+ </ul>
+ </footer>
+ </article>
+ </div>
+ </div>
<!-- Post -->
<div class="row">
<div class="callout">
diff --git a/output/feed.xml b/output/feed.xml
index fcdbe66..8a126ff 100644
--- a/output/feed.xml
+++ b/output/feed.xml
@@ -1,5 +1,21 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion
Blog</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Fri,
21 Mar 2025 00:00:00 +0000</lastBuildDate><item><title>Efficient Filter
Pushdown in
Parquet</title><link>https://datafusion.apache.org/blog/2025/03/21/parquet-pushdown</link><description><style>
+<rss version="2.0"><channel><title>Apache DataFusion
Blog</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Mon,
24 Mar 2025 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion
46.0.0
Released</title><link>https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0</link><description><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>We&rsquo;re excited to announce the release
of&nbsp;<strong>Apache DataFusion 46.0.0</strong>! This new
version represents a significant milestone for the project, packing in a wide
range of improvements and fixes. You can find the complete details in
the&nbsp;full <a
href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md">changelog</a>.
We&rsquo;ll highlight the most important changes below
…</p></descript [...]
figure {
margin: 20px 0;
}
diff --git a/output/feeds/all-en.atom.xml b/output/feeds/all-en.atom.xml
index 450d535..effa358 100644
--- a/output/feeds/all-en.atom.xml
+++ b/output/feeds/all-en.atom.xml
@@ -1,5 +1,107 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom";><title>Apache DataFusion
Blog</title><link href="https://datafusion.apache.org/blog/";
rel="alternate"></link><link
href="https://datafusion.apache.org/blog/feeds/all-en.atom.xml";
rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-03-21T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Efficient
Filter Pushdown in Parquet</title><link
href="https://datafusion.apache.org/blog/2025/03/21/parquet-pushdown";
rel="alter [...]
+<feed xmlns="http://www.w3.org/2005/Atom";><title>Apache DataFusion
Blog</title><link href="https://datafusion.apache.org/blog/";
rel="alternate"></link><link
href="https://datafusion.apache.org/blog/feeds/all-en.atom.xml";
rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-03-24T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache
DataFusion 46.0.0 Released</title><link
href="https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0";
rel="alterna [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>We&rsquo;re excited to announce the release
of&nbsp;<strong>Apache DataFusion 46.0.0</strong>! This new
version represents a significant milestone for the project, packing in a wide
range of improvements and fixes. You can find the complete details in
the&nbsp;full <a
href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md">changelog</a>.
We&rsquo;ll highlight the most important changes below
…</p></summary> [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>We&rsquo;re excited to announce the release
of&nbsp;<strong>Apache DataFusion 46.0.0</strong>! This new
version represents a significant milestone for the project, packing in a wide
range of improvements and fixes. You can find the complete details in
the&nbsp;full <a
href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md">changelog</a>.
We&rsquo;ll highlight the most important changes below and guide you
through [...]
+<h2>Breaking Changes</h2>
+<p>DataFusion 46.0.0 brings a few&nbsp;<strong>breaking
changes</strong>&nbsp;that may require adjustments to your code as
described in the <a
href="https://datafusion.apache.org/library-user-guide/upgrading.html">Upgrade
Guide</a>. Here are the most notable ones:</p>
+<ul>
+<li><a
href="https://github.com/apache/datafusion/pull/14224#">Unified
<code>DataSourceExec</code> Execution
Plan</a><strong>:</strong>&nbsp;DataFusion 46.0.0
introduces a major refactor of scan operators. The separate
file-format-specific execution plan nodes
(<code>ParquetExec</code>,&nbsp;<code>CsvExec</code>,&nbsp;<code>JsonExec</code>,&nbsp;<code>AvroExec</code>,
etc.) have been [...]
+<li><a
href="https://github.com/apache/arrow-datafusion/issues/7360#:~:text=2">**Error
Handling Improvements</a>
(<code>DataFusionError::Collection</code>):**&nbsp;We began
overhauling DataFusion&rsquo;s approach to error handling. In this release,
a new error
variant&nbsp;<code>DataFusionError::Collection</code>&nbsp;(and
related mechanisms) has been introduced to aggregate multiple errors into one.
This is part of a broader effo [...]
+</ul>
+<h2>Performance Improvements</h2>
+<p>DataFusion 46.0.0 comes with a slew of performance enhancements
across the board. Here are some of the noteworthy optimizations in this
release:</p>
+<ul>
+<li><strong>Faster&nbsp;<code>median()</code>&nbsp;(no
grouping):</strong>&nbsp;The&nbsp;<code>median()</code>&nbsp;aggregate
function got a special fast path when used without
a&nbsp;<code>GROUP BY</code>. By optimizing its accumulator,
median calculation is about&nbsp;<strong>2&times;
faster</strong>&nbsp;in the single-group case. If you
use&nbsp;<code>MEDIAN()</code>&a [...]
+<li><strong>Optimized&nbsp;<code>FIRST_VALUE</code>/<code>LAST_VALUE</code>:</strong>&nbsp;The&nbsp;<code>FIRST_VALUE</code>&nbsp;and&nbsp;<code>LAST_VALUE</code>&nbsp;window
functions have been improved by avoiding an internal sort of rows. Instead of
sorting each partition, the implementation now uses a direct approach to pick
the first/last element. This yields&nbsp;<strong>10&ndash [...]
+<li><strong><code>repeat()</code>&nbsp;String
Function Boost:</strong>&nbsp;Repeating strings is now more efficient
&ndash; the&nbsp;<code>repeat(text,
n)</code>&nbsp;function was optimized by
about&nbsp;<strong>50%</strong>. This was achieved by reducing
allocations and using a more efficient concatenation strategy. If you generate
large repeated strings in queries, this can cut the time nearly in half (PR <
[...]
+<li><strong>Ultra-fast&nbsp;<code>uuid()</code>&nbsp;UDF:</strong>&nbsp;The&nbsp;<code>uuid()</code>&nbsp;function
(which generates random UUID strings) received a major speed-up.
It&rsquo;s now roughly&nbsp;<strong>40&times;
faster</strong>&nbsp;than before! The new implementation avoids
unnecessary string copying and uses a more direct conversion to hex, making
bulk UUID generation far more practi [...]
+<li><strong>Accelerated&nbsp;<code>chr()</code>&nbsp;and&nbsp;<code>to_hex()</code>:</strong>&nbsp;Several
scalar functions have been micro-optimized.
The&nbsp;<code>chr()</code>&nbsp;function (which returns
the character for a given ASCII code) is
about&nbsp;<strong>4&times; faster</strong>&nbsp;now,
and the&nbsp;<code>to_hex()</code>&nbsp;function (which
converts nu [...]
+<li><strong>No More RowConverter in Grouped
Ordering:</strong>&nbsp;We removed an inefficient step in
the&nbsp;<em>partial grouping</em>&nbsp;algorithm.
The&nbsp;<code>GroupOrderingPartial</code>&nbsp;operator no
longer converts data to &ldquo;row format&rdquo; for each batch
(via&nbsp;<code>RowConverter</code>). Instead, it uses a direct
arrow-based approach to detect sort key changes. This eliminated [...]
+<li><strong>Predicate Pruning for&nbsp;<code>NOT
LIKE</code>:</strong>&nbsp;DataFusion&rsquo;s parquet
reader can now prune row groups using&nbsp;<code>NOT
LIKE</code>&nbsp;filters, similar to how it
handles&nbsp;<code>LIKE</code>. This means if you have a filter
such as&nbsp;<code>column NOT LIKE 'prefix%'</code>, DataFusion
can use min/max statistics to skip reading files/parts that can be det [...]
+</ul>
+<h2>Google Summer of Code 2025</h2>
+<p>Another exciting development:&nbsp;<strong>Apache
DataFusion has been accepted as a mentoring organization for Google Summer of
Code (GSoC) 2025</strong>! 🎉 This means that this summer, students from
around the world will have the opportunity to contribute to DataFusion under
the guidance of our committers. We have put together <a
href="https://datafusion.apache.org/contributor-guide/gsoc_project_ideas.html">a
list of project ideas</a> that candidates [...]
+<p>If you&rsquo;re interested, check out our&nbsp;<a
href="https://datafusion.apache.org/contributor-guide/gsoc_application_guidelines.html">GSoC
Application Guidelines</a>. We encourage students to reach out, discuss
ideas with us, and apply.</p>
+<h2>Highlighted New Features</h2>
+<h3>Improved Diagnostics</h3>
+<p>DataFusion 46.0.0 introduces a new&nbsp;<a
href="https://github.com/apache/datafusion/issues/14429"><strong>SQL
Diagnostics framework</strong></a>&nbsp;to make error messages
more understandable. This comes in the form of
new&nbsp;<code>Diagnostic</code>&nbsp;and&nbsp;<code>DiagnosticEntry</code>&nbsp;types,
which allow the system to attach rich context (like source query text spans)
to error messages. In pr [...]
+<p>For example, if you reference an unknown table or miss a column in
<code>GROUP BY</code> the error message will include the query
snippet causing the error. These diagnostics are meant for end-users of
applications built on DataFusion, providing clearer messages instead of generic
errors. Here&rsquo;s an example:</p>
+<p><img alt="diagnostic-example" class="img-responsive"
src="/blog/images/datafusion-46.0.0/diagnostic-example.png"
width="80%"/></p>
+<p>Currently, diagnostics cover unresolved table/column references,
missing <code>GROUP BY</code> columns, ambiguous references, wrong
number of UNION columns, type mismatches, and a few others. Future releases
will extend this to more error types. This feature should greatly ease
debugging of complex SQL by pinpointing errors directly in the query text. We
thank <a href="https://github.com/eliaperantoni">@eliaperantoni</a>
for his contributions in this proj [...]
+<h3>Unified&nbsp;<code>DataSourceExec</code>&nbsp;for
Table Providers</h3>
+<p>As mentioned, DataFusion now uses a
unified&nbsp;<code>DataSourceExec</code>&nbsp;for reading
tables, which is both a breaking change and a feature.&nbsp;<em>Why
is this important?</em>&nbsp;The new approach simplifies how custom
table providers are integrated and optimized. Namely, the optimizer can treat
file scans uniformly and push down filters/limits more consistently when there
is one execution plan that handles all data sources. The [...]
+<p>All built-in sources (Parquet, CSV, Avro, Arrow, JSON, etc.) have
been migrated to this framework. This unification makes the codebase cleaner
and sets the stage for future enhancements (like consistent metadata handling
and limit pushdown across all formats). Check out PR <a
href="https://github.com/apache/datafusion/pull/14224">#14224</a> for
design details. We thank <a
href="https://github.com/mertak-synnada">@mertak-synnada</a> and <a
href="https:/ [...]
+<h3>FFI Support for Scalar UDFs</h3>
+<p>DataFusion&rsquo;s Foreign Function Interface (FFI) has been
extended to support&nbsp;<a
href="https://github.com/apache/datafusion/pull/14579"><strong>user-defined
scalar functions</strong></a>&nbsp;defined in external
languages. In 46.0.0, you can now expose a custom scalar UDF through the FFI
layer and use it in DataFusion as if it were built-in. This is particularly
exciting for the <strong>Python bindings</strong> and other la [...]
+<h3>New Statistics/Distribution Framework</h3>
+<p>This release, thanks mainly to <a
href="https://github.com/Fly-Style">@Fly-Style</a> with contributions
from <a href="https://github.com/ozankabak">@ozankabak</a> and
<a href="https://github.com/berkaysynnada">@berkaysynnada</a>,
includes the initial pieces of a&nbsp;<a
href="https://github.com/apache/datafusion/pull/14699">**redesigned
statistics framework</a>.<strong> DataFusion&rsquo;s optimizer
can now represent c [...]
+<p>For example, if a filter expression is applied to a column with a
known uniform distribution range, the optimizer can propagate that to estimate
result selectivity more accurately. Similarly, comparisons
(<code>=</code>, <code>&gt;</code>, etc.) on
columns yield Bernoulli distributions (with true/false probabilities) in this
model.</p>
+<p>This is a foundational change with many follow-on PRs underway. Even
though the immediate user-visible effect is limited (the optimizer didn't
magically improve by an order of magnitude overnight), but it lays groundwork
for more advanced query planning in the future. Over time, as statistics
information encapsulated in <code>Distribution</code>s get
integrated, DataFusion will be able to make smarter decisions like more
aggressive parquet pruning, better join orderi [...]
+<h3>Aggregate Monotonicity and Window Ordering</h3>
+<p>DataFusion 46.0.0 adds a new concept of <a
href="https://github.com/apache/datafusion/pull/14271"><strong>set-monotonicity</strong></a>
for certain transformations, which helps avoid unnecessary sort operations. In
particular, the planner now understands when a <strong>window function
introduces new orderings of data</strong>.</p>
+<p>For example, DataFusion now recognizes that a window-aggregate like
<code>MAX</code> on a column can produce a result that is
<strong>monotonically increasing</strong>, even if the input column
is unordered &mdash; depending on the window frame used.</p>
+<p>Consider the following query:</p>
+<div
class="codehilite"><pre><span></span><code><span
class="k">SELECT</span> <span class="k">MAX</span><span
class="p">(</span><span class="n">c1</span><span
class="p">)</span> <span class="n">OVER</span> <span
class="p">(</span>
+ <span class="k">ROWS</span> <span
class="k">BETWEEN</span> <span class="n">UNBOUNDED</span>
<span class="n">PRECEDING</span> <span
class="k">AND</span> <span class="k">CURRENT</span>
<span class="k">ROW</span>
+<span class="p">)</span> <span class="k">AS</span>
<span class="n">max_c1</span>
+<span class="k">FROM</span> <span
class="n">c1_table</span>
+<span class="k">ORDER</span> <span class="k">BY</span>
<span class="n">max_c1</span><span class="p">;</span>
+</code></pre></div>
+<p>In earlier versions of DataFusion, this query would require an
additional SortExec on max_c1 to satisfy the ORDER BY clause. However, with the
new set-monotonicity logic, the planner knows that MAX(...) OVER (...) produces
values that are not smaller than the previous row, making the extra sort
redundant. This leads to more efficient query execution.</p>
+<p>PR <a
href="https://github.com/apache/datafusion/pull/14271">#14271</a>
introduced the core monotonicity tracking for aggregates and window functions.
+PR <a
href="https://github.com/apache/datafusion/pull/14813">#14813</a>
improved ordering preservation within various window frame types, and brought
an extensive test coverage.
+Huge thanks to <a
href="https://github.com/berkaysynnada">@berkaysynnada</a> and <a
href="https://github.com/mertak-synnada">@mertak-synnada</a> for
designing and implementing this optimizer enhancement!</p>
+<h3>UNION [ALL | DISTINCT] BY NAME Support</h3>
+<p>DataFusion now supports UNION BY NAME and UNION ALL BY NAME, which
align columns by name instead of position. This matches functionality found in
systems like Spark and DuckDB and simplifies combining heterogeneously ordered
result sets.</p>
+<p>You no longer need to rewrite column order manually &mdash; just
write:</p>
+<div
class="codehilite"><pre><span></span><code><span
class="k">SELECT</span> <span
class="n">col1</span><span class="p">,</span> <span
class="n">col2</span> <span class="k">FROM</span> <span
class="n">t1</span>
+<span class="k">UNION</span> <span
class="k">ALL</span> <span class="k">BY</span> <span
class="n">NAME</span>
+<span class="k">SELECT</span> <span
class="n">col2</span><span class="p">,</span> <span
class="n">col1</span> <span class="k">FROM</span> <span
class="n">t2</span><span class="p">;</span>
+</code></pre></div>
+<p>Under the hood, this is supported by the new union_by_name() and
union_by_name_distinct() plan builder methods.</p>
+<p>Thanks to <a
href="https://github.com/rkrishn7">@rkrishn7</a> for PR <a
href="https://github.com/apache/datafusion/pull/14538">#14538</a>.</p>
+<h3>New range() Table Function</h3>
+<p>A new table-valued function range(start, stop, step) has been added
to make it easy to generate integer sequences &mdash; similar to
PostgreSQL&rsquo;s generate_series() or Spark&rsquo;s range().</p>
+<p>Example:</p>
+<div
class="codehilite"><pre><span></span><code><span
class="k">SELECT</span> <span class="o">*</span> <span
class="k">FROM</span> <span class="n">range</span><span
class="p">(</span><span class="mi">1</span><span
class="p">,</span> <span class="mi">10</span><span
class="p">,</span> <span class="mi">2</span><span
class="p">);</span>
+</code></pre></div>
+<p>This returns: 1, 3, 5, 7, 9. It&rsquo;s great for testing, cross
joins, surrogate keys, and more.</p>
+<p>Thanks to <a
href="https://github.com/simonvandel">@simonvandel</a> for PR <a
href="https://github.com/apache/datafusion/pull/14830">#14830</a>.</p>
+<h2>Upgrade Guide and Changelog</h2>
+<p>Upgrading to 46.0.0 should be straightforward for most users, but do
review the&nbsp;<a
href="https://datafusion.apache.org/library-user-guide/upgrading.html">Upgrade
Guide for DataFusion 46.0.0</a>&nbsp;for detailed steps and code
changes. The upgrade guide covers the breaking changes mentioned (like
replacing old exec nodes with&nbsp;<code>DataSourceExec</code>,
updating UDF invocation to&nbsp;<code>invoke_with_args</code>,
e [...]
+<h2>Get Involved</h2>
+<p>Apache DataFusion is an open-source project, and we welcome
involvement from anyone interested. Now is a great time to take 46.0.0 for a
spin: try it out on your workloads, and let us know if you encounter any issues
or have suggestions. You can report bugs or request features on
our&nbsp;GitHub issue tracker, or better yet, submit a pull request. Join
our community discussions &ndash; whether you have questions, want to share
how you&rsquo;re using DataFusion, or ar [...]
+<p>Happy querying!</p></content><category
term="blog"></category></entry><entry><title>Efficient Filter Pushdown in
Parquet</title><link
href="https://datafusion.apache.org/blog/2025/03/21/parquet-pushdown";
rel="alternate"></link><published>2025-03-21T00:00:00+00:00</published><updated>2025-03-21T00:00:00+00:00</updated><author><name>Xiangpeng
Hao</name></author><id>tag:datafusion.apache.org,2025-03-21:/blog/2025/03/21/parquet-pushdown</id><summary
type="html"><style>
figure {
margin: 20px 0;
}
diff --git a/output/feeds/blog.atom.xml b/output/feeds/blog.atom.xml
index dbe61aa..30462c6 100644
--- a/output/feeds/blog.atom.xml
+++ b/output/feeds/blog.atom.xml
@@ -1,5 +1,107 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom";><title>Apache DataFusion Blog -
blog</title><link href="https://datafusion.apache.org/blog/";
rel="alternate"></link><link
href="https://datafusion.apache.org/blog/feeds/blog.atom.xml";
rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-03-21T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Efficient
Filter Pushdown in Parquet</title><link
href="https://datafusion.apache.org/blog/2025/03/21/parquet-pushdown"; rel="
[...]
+<feed xmlns="http://www.w3.org/2005/Atom";><title>Apache DataFusion Blog -
blog</title><link href="https://datafusion.apache.org/blog/";
rel="alternate"></link><link
href="https://datafusion.apache.org/blog/feeds/blog.atom.xml";
rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-03-24T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache
DataFusion 46.0.0 Released</title><link
href="https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0"; rel="al
[...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>We&rsquo;re excited to announce the release
of&nbsp;<strong>Apache DataFusion 46.0.0</strong>! This new
version represents a significant milestone for the project, packing in a wide
range of improvements and fixes. You can find the complete details in
the&nbsp;full <a
href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md">changelog</a>.
We&rsquo;ll highlight the most important changes below
…</p></summary> [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>We&rsquo;re excited to announce the release
of&nbsp;<strong>Apache DataFusion 46.0.0</strong>! This new
version represents a significant milestone for the project, packing in a wide
range of improvements and fixes. You can find the complete details in
the&nbsp;full <a
href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md">changelog</a>.
We&rsquo;ll highlight the most important changes below and guide you
through [...]
+<h2>Breaking Changes</h2>
+<p>DataFusion 46.0.0 brings a few&nbsp;<strong>breaking
changes</strong>&nbsp;that may require adjustments to your code as
described in the <a
href="https://datafusion.apache.org/library-user-guide/upgrading.html">Upgrade
Guide</a>. Here are the most notable ones:</p>
+<ul>
+<li><a
href="https://github.com/apache/datafusion/pull/14224#">Unified
<code>DataSourceExec</code> Execution
Plan</a><strong>:</strong>&nbsp;DataFusion 46.0.0
introduces a major refactor of scan operators. The separate
file-format-specific execution plan nodes
(<code>ParquetExec</code>,&nbsp;<code>CsvExec</code>,&nbsp;<code>JsonExec</code>,&nbsp;<code>AvroExec</code>,
etc.) have been [...]
+<li><a
href="https://github.com/apache/arrow-datafusion/issues/7360#:~:text=2">**Error
Handling Improvements</a>
(<code>DataFusionError::Collection</code>):**&nbsp;We began
overhauling DataFusion&rsquo;s approach to error handling. In this release,
a new error
variant&nbsp;<code>DataFusionError::Collection</code>&nbsp;(and
related mechanisms) has been introduced to aggregate multiple errors into one.
This is part of a broader effo [...]
+</ul>
+<h2>Performance Improvements</h2>
+<p>DataFusion 46.0.0 comes with a slew of performance enhancements
across the board. Here are some of the noteworthy optimizations in this
release:</p>
+<ul>
+<li><strong>Faster&nbsp;<code>median()</code>&nbsp;(no
grouping):</strong>&nbsp;The&nbsp;<code>median()</code>&nbsp;aggregate
function got a special fast path when used without
a&nbsp;<code>GROUP BY</code>. By optimizing its accumulator,
median calculation is about&nbsp;<strong>2&times;
faster</strong>&nbsp;in the single-group case. If you
use&nbsp;<code>MEDIAN()</code>&a [...]
+<li><strong>Optimized&nbsp;<code>FIRST_VALUE</code>/<code>LAST_VALUE</code>:</strong>&nbsp;The&nbsp;<code>FIRST_VALUE</code>&nbsp;and&nbsp;<code>LAST_VALUE</code>&nbsp;window
functions have been improved by avoiding an internal sort of rows. Instead of
sorting each partition, the implementation now uses a direct approach to pick
the first/last element. This yields&nbsp;<strong>10&ndash [...]
+<li><strong><code>repeat()</code>&nbsp;String
Function Boost:</strong>&nbsp;Repeating strings is now more efficient
&ndash; the&nbsp;<code>repeat(text,
n)</code>&nbsp;function was optimized by
about&nbsp;<strong>50%</strong>. This was achieved by reducing
allocations and using a more efficient concatenation strategy. If you generate
large repeated strings in queries, this can cut the time nearly in half (PR <
[...]
+<li><strong>Ultra-fast&nbsp;<code>uuid()</code>&nbsp;UDF:</strong>&nbsp;The&nbsp;<code>uuid()</code>&nbsp;function
(which generates random UUID strings) received a major speed-up.
It&rsquo;s now roughly&nbsp;<strong>40&times;
faster</strong>&nbsp;than before! The new implementation avoids
unnecessary string copying and uses a more direct conversion to hex, making
bulk UUID generation far more practi [...]
+<li><strong>Accelerated&nbsp;<code>chr()</code>&nbsp;and&nbsp;<code>to_hex()</code>:</strong>&nbsp;Several
scalar functions have been micro-optimized.
The&nbsp;<code>chr()</code>&nbsp;function (which returns
the character for a given ASCII code) is
about&nbsp;<strong>4&times; faster</strong>&nbsp;now,
and the&nbsp;<code>to_hex()</code>&nbsp;function (which
converts nu [...]
+<li><strong>No More RowConverter in Grouped
Ordering:</strong>&nbsp;We removed an inefficient step in
the&nbsp;<em>partial grouping</em>&nbsp;algorithm.
The&nbsp;<code>GroupOrderingPartial</code>&nbsp;operator no
longer converts data to &ldquo;row format&rdquo; for each batch
(via&nbsp;<code>RowConverter</code>). Instead, it uses a direct
arrow-based approach to detect sort key changes. This eliminated [...]
+<li><strong>Predicate Pruning for&nbsp;<code>NOT
LIKE</code>:</strong>&nbsp;DataFusion&rsquo;s parquet
reader can now prune row groups using&nbsp;<code>NOT
LIKE</code>&nbsp;filters, similar to how it
handles&nbsp;<code>LIKE</code>. This means if you have a filter
such as&nbsp;<code>column NOT LIKE 'prefix%'</code>, DataFusion
can use min/max statistics to skip reading files/parts that can be det [...]
+</ul>
+<h2>Google Summer of Code 2025</h2>
+<p>Another exciting development:&nbsp;<strong>Apache
DataFusion has been accepted as a mentoring organization for Google Summer of
Code (GSoC) 2025</strong>! 🎉 This means that this summer, students from
around the world will have the opportunity to contribute to DataFusion under
the guidance of our committers. We have put together <a
href="https://datafusion.apache.org/contributor-guide/gsoc_project_ideas.html">a
list of project ideas</a> that candidates [...]
+<p>If you&rsquo;re interested, check out our&nbsp;<a
href="https://datafusion.apache.org/contributor-guide/gsoc_application_guidelines.html">GSoC
Application Guidelines</a>. We encourage students to reach out, discuss
ideas with us, and apply.</p>
+<h2>Highlighted New Features</h2>
+<h3>Improved Diagnostics</h3>
+<p>DataFusion 46.0.0 introduces a new&nbsp;<a
href="https://github.com/apache/datafusion/issues/14429"><strong>SQL
Diagnostics framework</strong></a>&nbsp;to make error messages
more understandable. This comes in the form of
new&nbsp;<code>Diagnostic</code>&nbsp;and&nbsp;<code>DiagnosticEntry</code>&nbsp;types,
which allow the system to attach rich context (like source query text spans)
to error messages. In pr [...]
+<p>For example, if you reference an unknown table or miss a column in
<code>GROUP BY</code> the error message will include the query
snippet causing the error. These diagnostics are meant for end-users of
applications built on DataFusion, providing clearer messages instead of generic
errors. Here&rsquo;s an example:</p>
+<p><img alt="diagnostic-example" class="img-responsive"
src="/blog/images/datafusion-46.0.0/diagnostic-example.png"
width="80%"/></p>
+<p>Currently, diagnostics cover unresolved table/column references,
missing <code>GROUP BY</code> columns, ambiguous references, wrong
number of UNION columns, type mismatches, and a few others. Future releases
will extend this to more error types. This feature should greatly ease
debugging of complex SQL by pinpointing errors directly in the query text. We
thank <a href="https://github.com/eliaperantoni">@eliaperantoni</a>
for his contributions in this proj [...]
+<h3>Unified&nbsp;<code>DataSourceExec</code>&nbsp;for
Table Providers</h3>
+<p>As mentioned, DataFusion now uses a
unified&nbsp;<code>DataSourceExec</code>&nbsp;for reading
tables, which is both a breaking change and a feature.&nbsp;<em>Why
is this important?</em>&nbsp;The new approach simplifies how custom
table providers are integrated and optimized. Namely, the optimizer can treat
file scans uniformly and push down filters/limits more consistently when there
is one execution plan that handles all data sources. The [...]
+<p>All built-in sources (Parquet, CSV, Avro, Arrow, JSON, etc.) have
been migrated to this framework. This unification makes the codebase cleaner
and sets the stage for future enhancements (like consistent metadata handling
and limit pushdown across all formats). Check out PR <a
href="https://github.com/apache/datafusion/pull/14224">#14224</a> for
design details. We thank <a
href="https://github.com/mertak-synnada">@mertak-synnada</a> and <a
href="https:/ [...]
+<h3>FFI Support for Scalar UDFs</h3>
+<p>DataFusion&rsquo;s Foreign Function Interface (FFI) has been
extended to support&nbsp;<a
href="https://github.com/apache/datafusion/pull/14579"><strong>user-defined
scalar functions</strong></a>&nbsp;defined in external
languages. In 46.0.0, you can now expose a custom scalar UDF through the FFI
layer and use it in DataFusion as if it were built-in. This is particularly
exciting for the <strong>Python bindings</strong> and other la [...]
+<h3>New Statistics/Distribution Framework</h3>
+<p>This release, thanks mainly to <a
href="https://github.com/Fly-Style">@Fly-Style</a> with contributions
from <a href="https://github.com/ozankabak">@ozankabak</a> and
<a href="https://github.com/berkaysynnada">@berkaysynnada</a>,
includes the initial pieces of a&nbsp;<a
href="https://github.com/apache/datafusion/pull/14699">**redesigned
statistics framework</a>.<strong> DataFusion&rsquo;s optimizer
can now represent c [...]
+<p>For example, if a filter expression is applied to a column with a
known uniform distribution range, the optimizer can propagate that to estimate
result selectivity more accurately. Similarly, comparisons
(<code>=</code>, <code>&gt;</code>, etc.) on
columns yield Bernoulli distributions (with true/false probabilities) in this
model.</p>
+<p>This is a foundational change with many follow-on PRs underway. Even
though the immediate user-visible effect is limited (the optimizer didn't
magically improve by an order of magnitude overnight), but it lays groundwork
for more advanced query planning in the future. Over time, as statistics
information encapsulated in <code>Distribution</code>s get
integrated, DataFusion will be able to make smarter decisions like more
aggressive parquet pruning, better join orderi [...]
+<h3>Aggregate Monotonicity and Window Ordering</h3>
+<p>DataFusion 46.0.0 adds a new concept of <a
href="https://github.com/apache/datafusion/pull/14271"><strong>set-monotonicity</strong></a>
for certain transformations, which helps avoid unnecessary sort operations. In
particular, the planner now understands when a <strong>window function
introduces new orderings of data</strong>.</p>
+<p>For example, DataFusion now recognizes that a window-aggregate like
<code>MAX</code> on a column can produce a result that is
<strong>monotonically increasing</strong>, even if the input column
is unordered &mdash; depending on the window frame used.</p>
+<p>Consider the following query:</p>
+<div
class="codehilite"><pre><span></span><code><span
class="k">SELECT</span> <span class="k">MAX</span><span
class="p">(</span><span class="n">c1</span><span
class="p">)</span> <span class="n">OVER</span> <span
class="p">(</span>
+ <span class="k">ROWS</span> <span
class="k">BETWEEN</span> <span class="n">UNBOUNDED</span>
<span class="n">PRECEDING</span> <span
class="k">AND</span> <span class="k">CURRENT</span>
<span class="k">ROW</span>
+<span class="p">)</span> <span class="k">AS</span>
<span class="n">max_c1</span>
+<span class="k">FROM</span> <span
class="n">c1_table</span>
+<span class="k">ORDER</span> <span class="k">BY</span>
<span class="n">max_c1</span><span class="p">;</span>
+</code></pre></div>
+<p>In earlier versions of DataFusion, this query would require an
additional SortExec on max_c1 to satisfy the ORDER BY clause. However, with the
new set-monotonicity logic, the planner knows that MAX(...) OVER (...) produces
values that are not smaller than the previous row, making the extra sort
redundant. This leads to more efficient query execution.</p>
+<p>PR <a
href="https://github.com/apache/datafusion/pull/14271">#14271</a>
introduced the core monotonicity tracking for aggregates and window functions.
+PR <a
href="https://github.com/apache/datafusion/pull/14813">#14813</a>
improved ordering preservation within various window frame types, and brought
an extensive test coverage.
+Huge thanks to <a
href="https://github.com/berkaysynnada">@berkaysynnada</a> and <a
href="https://github.com/mertak-synnada">@mertak-synnada</a> for
designing and implementing this optimizer enhancement!</p>
+<h3>UNION [ALL | DISTINCT] BY NAME Support</h3>
+<p>DataFusion now supports UNION BY NAME and UNION ALL BY NAME, which
align columns by name instead of position. This matches functionality found in
systems like Spark and DuckDB and simplifies combining heterogeneously ordered
result sets.</p>
+<p>You no longer need to rewrite column order manually &mdash; just
write:</p>
+<div
class="codehilite"><pre><span></span><code><span
class="k">SELECT</span> <span
class="n">col1</span><span class="p">,</span> <span
class="n">col2</span> <span class="k">FROM</span> <span
class="n">t1</span>
+<span class="k">UNION</span> <span
class="k">ALL</span> <span class="k">BY</span> <span
class="n">NAME</span>
+<span class="k">SELECT</span> <span
class="n">col2</span><span class="p">,</span> <span
class="n">col1</span> <span class="k">FROM</span> <span
class="n">t2</span><span class="p">;</span>
+</code></pre></div>
+<p>Under the hood, this is supported by the new union_by_name() and
union_by_name_distinct() plan builder methods.</p>
+<p>Thanks to <a
href="https://github.com/rkrishn7">@rkrishn7</a> for PR <a
href="https://github.com/apache/datafusion/pull/14538">#14538</a>.</p>
+<h3>New range() Table Function</h3>
+<p>A new table-valued function range(start, stop, step) has been added
to make it easy to generate integer sequences &mdash; similar to
PostgreSQL&rsquo;s generate_series() or Spark&rsquo;s range().</p>
+<p>Example:</p>
+<div
class="codehilite"><pre><span></span><code><span
class="k">SELECT</span> <span class="o">*</span> <span
class="k">FROM</span> <span class="n">range</span><span
class="p">(</span><span class="mi">1</span><span
class="p">,</span> <span class="mi">10</span><span
class="p">,</span> <span class="mi">2</span><span
class="p">);</span>
+</code></pre></div>
+<p>This returns: 1, 3, 5, 7, 9. It&rsquo;s great for testing, cross
joins, surrogate keys, and more.</p>
+<p>Thanks to <a
href="https://github.com/simonvandel">@simonvandel</a> for PR <a
href="https://github.com/apache/datafusion/pull/14830">#14830</a>.</p>
+<h2>Upgrade Guide and Changelog</h2>
+<p>Upgrading to 46.0.0 should be straightforward for most users, but do
review the&nbsp;<a
href="https://datafusion.apache.org/library-user-guide/upgrading.html">Upgrade
Guide for DataFusion 46.0.0</a>&nbsp;for detailed steps and code
changes. The upgrade guide covers the breaking changes mentioned (like
replacing old exec nodes with&nbsp;<code>DataSourceExec</code>,
updating UDF invocation to&nbsp;<code>invoke_with_args</code>,
e [...]
+<h2>Get Involved</h2>
+<p>Apache DataFusion is an open-source project, and we welcome
involvement from anyone interested. Now is a great time to take 46.0.0 for a
spin: try it out on your workloads, and let us know if you encounter any issues
or have suggestions. You can report bugs or request features on
our&nbsp;GitHub issue tracker, or better yet, submit a pull request. Join
our community discussions &ndash; whether you have questions, want to share
how you&rsquo;re using DataFusion, or ar [...]
+<p>Happy querying!</p></content><category
term="blog"></category></entry><entry><title>Efficient Filter Pushdown in
Parquet</title><link
href="https://datafusion.apache.org/blog/2025/03/21/parquet-pushdown";
rel="alternate"></link><published>2025-03-21T00:00:00+00:00</published><updated>2025-03-21T00:00:00+00:00</updated><author><name>Xiangpeng
Hao</name></author><id>tag:datafusion.apache.org,2025-03-21:/blog/2025/03/21/parquet-pushdown</id><summary
type="html"><style>
figure {
margin: 20px 0;
}
diff --git
a/output/feeds/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.atom.xml
b/output/feeds/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.atom.xml
new file mode 100644
index 0000000..516826e
--- /dev/null
+++ b/output/feeds/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.atom.xml
@@ -0,0 +1,104 @@
+<?xml version="1.0" encoding="utf-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom";><title>Apache DataFusion Blog -
Oznur Hanci and Berkay Sahin on behalf of the PMC</title><link
href="https://datafusion.apache.org/blog/"; rel="alternate"></link><link
href="https://datafusion.apache.org/blog/feeds/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.atom.xml";
rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-03-24T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Apache
DataFusion 46.0.0 Released</ [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>We&rsquo;re excited to announce the release
of&nbsp;<strong>Apache DataFusion 46.0.0</strong>! This new
version represents a significant milestone for the project, packing in a wide
range of improvements and fixes. You can find the complete details in
the&nbsp;full <a
href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md">changelog</a>.
We&rsquo;ll highlight the most important changes below
…</p></summary> [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>We&rsquo;re excited to announce the release
of&nbsp;<strong>Apache DataFusion 46.0.0</strong>! This new
version represents a significant milestone for the project, packing in a wide
range of improvements and fixes. You can find the complete details in
the&nbsp;full <a
href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md">changelog</a>.
We&rsquo;ll highlight the most important changes below and guide you
through [...]
+<h2>Breaking Changes</h2>
+<p>DataFusion 46.0.0 brings a few&nbsp;<strong>breaking
changes</strong>&nbsp;that may require adjustments to your code as
described in the <a
href="https://datafusion.apache.org/library-user-guide/upgrading.html">Upgrade
Guide</a>. Here are the most notable ones:</p>
+<ul>
+<li><a
href="https://github.com/apache/datafusion/pull/14224#">Unified
<code>DataSourceExec</code> Execution
Plan</a><strong>:</strong>&nbsp;DataFusion 46.0.0
introduces a major refactor of scan operators. The separate
file-format-specific execution plan nodes
(<code>ParquetExec</code>,&nbsp;<code>CsvExec</code>,&nbsp;<code>JsonExec</code>,&nbsp;<code>AvroExec</code>,
etc.) have been [...]
+<li><a
href="https://github.com/apache/arrow-datafusion/issues/7360#:~:text=2">**Error
Handling Improvements</a>
(<code>DataFusionError::Collection</code>):**&nbsp;We began
overhauling DataFusion&rsquo;s approach to error handling. In this release,
a new error
variant&nbsp;<code>DataFusionError::Collection</code>&nbsp;(and
related mechanisms) has been introduced to aggregate multiple errors into one.
This is part of a broader effo [...]
+</ul>
+<h2>Performance Improvements</h2>
+<p>DataFusion 46.0.0 comes with a slew of performance enhancements
across the board. Here are some of the noteworthy optimizations in this
release:</p>
+<ul>
+<li><strong>Faster&nbsp;<code>median()</code>&nbsp;(no
grouping):</strong>&nbsp;The&nbsp;<code>median()</code>&nbsp;aggregate
function got a special fast path when used without
a&nbsp;<code>GROUP BY</code>. By optimizing its accumulator,
median calculation is about&nbsp;<strong>2&times;
faster</strong>&nbsp;in the single-group case. If you
use&nbsp;<code>MEDIAN()</code>&a [...]
+<li><strong>Optimized&nbsp;<code>FIRST_VALUE</code>/<code>LAST_VALUE</code>:</strong>&nbsp;The&nbsp;<code>FIRST_VALUE</code>&nbsp;and&nbsp;<code>LAST_VALUE</code>&nbsp;window
functions have been improved by avoiding an internal sort of rows. Instead of
sorting each partition, the implementation now uses a direct approach to pick
the first/last element. This yields&nbsp;<strong>10&ndash [...]
+<li><strong><code>repeat()</code>&nbsp;String
Function Boost:</strong>&nbsp;Repeating strings is now more efficient
&ndash; the&nbsp;<code>repeat(text,
n)</code>&nbsp;function was optimized by
about&nbsp;<strong>50%</strong>. This was achieved by reducing
allocations and using a more efficient concatenation strategy. If you generate
large repeated strings in queries, this can cut the time nearly in half (PR <
[...]
+<li><strong>Ultra-fast&nbsp;<code>uuid()</code>&nbsp;UDF:</strong>&nbsp;The&nbsp;<code>uuid()</code>&nbsp;function
(which generates random UUID strings) received a major speed-up.
It&rsquo;s now roughly&nbsp;<strong>40&times;
faster</strong>&nbsp;than before! The new implementation avoids
unnecessary string copying and uses a more direct conversion to hex, making
bulk UUID generation far more practi [...]
+<li><strong>Accelerated&nbsp;<code>chr()</code>&nbsp;and&nbsp;<code>to_hex()</code>:</strong>&nbsp;Several
scalar functions have been micro-optimized.
The&nbsp;<code>chr()</code>&nbsp;function (which returns
the character for a given ASCII code) is
about&nbsp;<strong>4&times; faster</strong>&nbsp;now,
and the&nbsp;<code>to_hex()</code>&nbsp;function (which
converts nu [...]
+<li><strong>No More RowConverter in Grouped
Ordering:</strong>&nbsp;We removed an inefficient step in
the&nbsp;<em>partial grouping</em>&nbsp;algorithm.
The&nbsp;<code>GroupOrderingPartial</code>&nbsp;operator no
longer converts data to &ldquo;row format&rdquo; for each batch
(via&nbsp;<code>RowConverter</code>). Instead, it uses a direct
arrow-based approach to detect sort key changes. This eliminated [...]
+<li><strong>Predicate Pruning for&nbsp;<code>NOT
LIKE</code>:</strong>&nbsp;DataFusion&rsquo;s parquet
reader can now prune row groups using&nbsp;<code>NOT
LIKE</code>&nbsp;filters, similar to how it
handles&nbsp;<code>LIKE</code>. This means if you have a filter
such as&nbsp;<code>column NOT LIKE 'prefix%'</code>, DataFusion
can use min/max statistics to skip reading files/parts that can be det [...]
+</ul>
+<h2>Google Summer of Code 2025</h2>
+<p>Another exciting development:&nbsp;<strong>Apache
DataFusion has been accepted as a mentoring organization for Google Summer of
Code (GSoC) 2025</strong>! 🎉 This means that this summer, students from
around the world will have the opportunity to contribute to DataFusion under
the guidance of our committers. We have put together <a
href="https://datafusion.apache.org/contributor-guide/gsoc_project_ideas.html">a
list of project ideas</a> that candidates [...]
+<p>If you&rsquo;re interested, check out our&nbsp;<a
href="https://datafusion.apache.org/contributor-guide/gsoc_application_guidelines.html">GSoC
Application Guidelines</a>. We encourage students to reach out, discuss
ideas with us, and apply.</p>
+<h2>Highlighted New Features</h2>
+<h3>Improved Diagnostics</h3>
+<p>DataFusion 46.0.0 introduces a new&nbsp;<a
href="https://github.com/apache/datafusion/issues/14429"><strong>SQL
Diagnostics framework</strong></a>&nbsp;to make error messages
more understandable. This comes in the form of
new&nbsp;<code>Diagnostic</code>&nbsp;and&nbsp;<code>DiagnosticEntry</code>&nbsp;types,
which allow the system to attach rich context (like source query text spans)
to error messages. In pr [...]
+<p>For example, if you reference an unknown table or miss a column in
<code>GROUP BY</code> the error message will include the query
snippet causing the error. These diagnostics are meant for end-users of
applications built on DataFusion, providing clearer messages instead of generic
errors. Here&rsquo;s an example:</p>
+<p><img alt="diagnostic-example" class="img-responsive"
src="/blog/images/datafusion-46.0.0/diagnostic-example.png"
width="80%"/></p>
+<p>Currently, diagnostics cover unresolved table/column references,
missing <code>GROUP BY</code> columns, ambiguous references, wrong
number of UNION columns, type mismatches, and a few others. Future releases
will extend this to more error types. This feature should greatly ease
debugging of complex SQL by pinpointing errors directly in the query text. We
thank <a href="https://github.com/eliaperantoni">@eliaperantoni</a>
for his contributions in this proj [...]
+<h3>Unified&nbsp;<code>DataSourceExec</code>&nbsp;for
Table Providers</h3>
+<p>As mentioned, DataFusion now uses a
unified&nbsp;<code>DataSourceExec</code>&nbsp;for reading
tables, which is both a breaking change and a feature.&nbsp;<em>Why
is this important?</em>&nbsp;The new approach simplifies how custom
table providers are integrated and optimized. Namely, the optimizer can treat
file scans uniformly and push down filters/limits more consistently when there
is one execution plan that handles all data sources. The [...]
+<p>All built-in sources (Parquet, CSV, Avro, Arrow, JSON, etc.) have
been migrated to this framework. This unification makes the codebase cleaner
and sets the stage for future enhancements (like consistent metadata handling
and limit pushdown across all formats). Check out PR <a
href="https://github.com/apache/datafusion/pull/14224">#14224</a> for
design details. We thank <a
href="https://github.com/mertak-synnada">@mertak-synnada</a> and <a
href="https:/ [...]
+<h3>FFI Support for Scalar UDFs</h3>
+<p>DataFusion&rsquo;s Foreign Function Interface (FFI) has been
extended to support&nbsp;<a
href="https://github.com/apache/datafusion/pull/14579"><strong>user-defined
scalar functions</strong></a>&nbsp;defined in external
languages. In 46.0.0, you can now expose a custom scalar UDF through the FFI
layer and use it in DataFusion as if it were built-in. This is particularly
exciting for the <strong>Python bindings</strong> and other la [...]
+<h3>New Statistics/Distribution Framework</h3>
+<p>This release, thanks mainly to <a
href="https://github.com/Fly-Style">@Fly-Style</a> with contributions
from <a href="https://github.com/ozankabak">@ozankabak</a> and
<a href="https://github.com/berkaysynnada">@berkaysynnada</a>,
includes the initial pieces of a&nbsp;<a
href="https://github.com/apache/datafusion/pull/14699">**redesigned
statistics framework</a>.<strong> DataFusion&rsquo;s optimizer
can now represent c [...]
+<p>For example, if a filter expression is applied to a column with a
known uniform distribution range, the optimizer can propagate that to estimate
result selectivity more accurately. Similarly, comparisons
(<code>=</code>, <code>&gt;</code>, etc.) on
columns yield Bernoulli distributions (with true/false probabilities) in this
model.</p>
+<p>This is a foundational change with many follow-on PRs underway. Even
though the immediate user-visible effect is limited (the optimizer didn't
magically improve by an order of magnitude overnight), but it lays groundwork
for more advanced query planning in the future. Over time, as statistics
information encapsulated in <code>Distribution</code>s get
integrated, DataFusion will be able to make smarter decisions like more
aggressive parquet pruning, better join orderi [...]
+<h3>Aggregate Monotonicity and Window Ordering</h3>
+<p>DataFusion 46.0.0 adds a new concept of <a
href="https://github.com/apache/datafusion/pull/14271"><strong>set-monotonicity</strong></a>
for certain transformations, which helps avoid unnecessary sort operations. In
particular, the planner now understands when a <strong>window function
introduces new orderings of data</strong>.</p>
+<p>For example, DataFusion now recognizes that a window-aggregate like
<code>MAX</code> on a column can produce a result that is
<strong>monotonically increasing</strong>, even if the input column
is unordered &mdash; depending on the window frame used.</p>
+<p>Consider the following query:</p>
+<div
class="codehilite"><pre><span></span><code><span
class="k">SELECT</span> <span class="k">MAX</span><span
class="p">(</span><span class="n">c1</span><span
class="p">)</span> <span class="n">OVER</span> <span
class="p">(</span>
+ <span class="k">ROWS</span> <span
class="k">BETWEEN</span> <span class="n">UNBOUNDED</span>
<span class="n">PRECEDING</span> <span
class="k">AND</span> <span class="k">CURRENT</span>
<span class="k">ROW</span>
+<span class="p">)</span> <span class="k">AS</span>
<span class="n">max_c1</span>
+<span class="k">FROM</span> <span
class="n">c1_table</span>
+<span class="k">ORDER</span> <span class="k">BY</span>
<span class="n">max_c1</span><span class="p">;</span>
+</code></pre></div>
+<p>In earlier versions of DataFusion, this query would require an
additional SortExec on max_c1 to satisfy the ORDER BY clause. However, with the
new set-monotonicity logic, the planner knows that MAX(...) OVER (...) produces
values that are not smaller than the previous row, making the extra sort
redundant. This leads to more efficient query execution.</p>
+<p>PR <a
href="https://github.com/apache/datafusion/pull/14271">#14271</a>
introduced the core monotonicity tracking for aggregates and window functions.
+PR <a
href="https://github.com/apache/datafusion/pull/14813">#14813</a>
improved ordering preservation within various window frame types, and brought
an extensive test coverage.
+Huge thanks to <a
href="https://github.com/berkaysynnada">@berkaysynnada</a> and <a
href="https://github.com/mertak-synnada">@mertak-synnada</a> for
designing and implementing this optimizer enhancement!</p>
+<h3>UNION [ALL | DISTINCT] BY NAME Support</h3>
+<p>DataFusion now supports UNION BY NAME and UNION ALL BY NAME, which
align columns by name instead of position. This matches functionality found in
systems like Spark and DuckDB and simplifies combining heterogeneously ordered
result sets.</p>
+<p>You no longer need to rewrite column order manually &mdash; just
write:</p>
+<div
class="codehilite"><pre><span></span><code><span
class="k">SELECT</span> <span
class="n">col1</span><span class="p">,</span> <span
class="n">col2</span> <span class="k">FROM</span> <span
class="n">t1</span>
+<span class="k">UNION</span> <span
class="k">ALL</span> <span class="k">BY</span> <span
class="n">NAME</span>
+<span class="k">SELECT</span> <span
class="n">col2</span><span class="p">,</span> <span
class="n">col1</span> <span class="k">FROM</span> <span
class="n">t2</span><span class="p">;</span>
+</code></pre></div>
+<p>Under the hood, this is supported by the new union_by_name() and
union_by_name_distinct() plan builder methods.</p>
+<p>Thanks to <a
href="https://github.com/rkrishn7">@rkrishn7</a> for PR <a
href="https://github.com/apache/datafusion/pull/14538">#14538</a>.</p>
+<h3>New range() Table Function</h3>
+<p>A new table-valued function range(start, stop, step) has been added
to make it easy to generate integer sequences &mdash; similar to
PostgreSQL&rsquo;s generate_series() or Spark&rsquo;s range().</p>
+<p>Example:</p>
+<div
class="codehilite"><pre><span></span><code><span
class="k">SELECT</span> <span class="o">*</span> <span
class="k">FROM</span> <span class="n">range</span><span
class="p">(</span><span class="mi">1</span><span
class="p">,</span> <span class="mi">10</span><span
class="p">,</span> <span class="mi">2</span><span
class="p">);</span>
+</code></pre></div>
+<p>This returns: 1, 3, 5, 7, 9. It&rsquo;s great for testing, cross
joins, surrogate keys, and more.</p>
+<p>Thanks to <a
href="https://github.com/simonvandel">@simonvandel</a> for PR <a
href="https://github.com/apache/datafusion/pull/14830">#14830</a>.</p>
+<h2>Upgrade Guide and Changelog</h2>
+<p>Upgrading to 46.0.0 should be straightforward for most users, but do
review the&nbsp;<a
href="https://datafusion.apache.org/library-user-guide/upgrading.html">Upgrade
Guide for DataFusion 46.0.0</a>&nbsp;for detailed steps and code
changes. The upgrade guide covers the breaking changes mentioned (like
replacing old exec nodes with&nbsp;<code>DataSourceExec</code>,
updating UDF invocation to&nbsp;<code>invoke_with_args</code>,
e [...]
+<h2>Get Involved</h2>
+<p>Apache DataFusion is an open-source project, and we welcome
involvement from anyone interested. Now is a great time to take 46.0.0 for a
spin: try it out on your workloads, and let us know if you encounter any issues
or have suggestions. You can report bugs or request features on
our&nbsp;GitHub issue tracker, or better yet, submit a pull request. Join
our community discussions &ndash; whether you have questions, want to share
how you&rsquo;re using DataFusion, or ar [...]
+<p>Happy querying!</p></content><category
term="blog"></category></entry></feed>
\ No newline at end of file
diff --git
a/output/feeds/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.rss.xml
b/output/feeds/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.rss.xml
new file mode 100644
index 0000000..177a08c
--- /dev/null
+++ b/output/feeds/oznur-hanci-and-berkay-sahin-on-behalf-of-the-pmc.rss.xml
@@ -0,0 +1,18 @@
+<?xml version="1.0" encoding="utf-8"?>
+<rss version="2.0"><channel><title>Apache DataFusion Blog - Oznur Hanci and
Berkay Sahin on behalf of the
PMC</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Mon,
24 Mar 2025 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion
46.0.0
Released</title><link>https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0</link><description><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>We&rsquo;re excited to announce the release
of&nbsp;<strong>Apache DataFusion 46.0.0</strong>! This new
version represents a significant milestone for the project, packing in a wide
range of improvements and fixes. You can find the complete details in
the&nbsp;full <a
href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md">changelog</a>.
We&rsquo;ll highlight the most important changes below
…</p></descript [...]
\ No newline at end of file
diff --git a/output/images/datafusion-46.0.0/diagnostic-example.png
b/output/images/datafusion-46.0.0/diagnostic-example.png
new file mode 100644
index 0000000..0dc3df5
Binary files /dev/null and
b/output/images/datafusion-46.0.0/diagnostic-example.png differ
diff --git a/output/index.html b/output/index.html
index ce50f7d..306c43c 100644
--- a/output/index.html
+++ b/output/index.html
@@ -44,6 +44,41 @@
<p><i>Here you can find the latest updates from DataFusion and
related projects.</i></p>
+ <!-- Post -->
+ <div class="row">
+ <div class="callout">
+ <article class="post">
+ <header>
+ <div class="title">
+ <h1><a
href="/blog/2025/03/24/datafusion-46.0.0">Apache DataFusion 46.0.0
Released</a></h1>
+ <p>Posted on: Mon 24 March 2025 by Oznur Hanci and
Berkay Sahin on behalf of the PMC</p>
+ <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+<p>We’re excited to announce the release of <strong>Apache
DataFusion 46.0.0</strong>! This new version represents a significant milestone
for the project, packing in a wide range of improvements and fixes. You can
find the complete details in the full <a
href="https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md";>changelog</a>.
We’ll highlight the most important changes below …</p></p>
+ <footer>
+ <ul class="actions">
+ <div style="text-align: right"><a
href="/blog/2025/03/24/datafusion-46.0.0" class="button medium">Continue
Reading</a></div>
+ </ul>
+ <ul class="stats">
+ </ul>
+ </footer>
+ </article>
+ </div>
+ </div>
<!-- Post -->
<div class="row">
<div class="callout">
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
