This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 898789275f Publish built docs triggered by
d3792410d9444afe9eb6d67d383210f81dad3cdf
898789275f is described below
commit 898789275ff4c7c97d6b1b2ef835ca6221e12e29
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Mon May 18 21:55:07 2026 +0000
Publish built docs triggered by d3792410d9444afe9eb6d67d383210f81dad3cdf
---
.../library-user-guide/upgrading/54.0.0.md.txt | 39 ++++++++++++++++++++++
_sources/user-guide/configs.md.txt | 17 +++++-----
library-user-guide/upgrading/54.0.0.html | 29 ++++++++++++++++
searchindex.js | 2 +-
user-guide/configs.html | 16 +++++----
5 files changed, 88 insertions(+), 15 deletions(-)
diff --git a/_sources/library-user-guide/upgrading/54.0.0.md.txt
b/_sources/library-user-guide/upgrading/54.0.0.md.txt
index 0117a776b2..076b65444f 100644
--- a/_sources/library-user-guide/upgrading/54.0.0.md.txt
+++ b/_sources/library-user-guide/upgrading/54.0.0.md.txt
@@ -716,3 +716,42 @@ move the borrow behind an interior-mutability primitive
such as `Arc<Mutex<_>>`
or `Arc<OnceLock<_>>`.
See [PR #21803](https://github.com/apache/datafusion/pull/21803) for details.
+
+[20047]: https://github.com/apache/datafusion/pull/20047
+
+### File statistics cache is now memory-limited and managed by the
`CacheManager`
+
+The file statistics cache used by `ListingTable` is now memory-limited and
+centrally managed through the `CacheManager`.
+
+To configure the cache size use the `file_statistics_cache_limit` setting:
+
+```sql
+SET datafusion.runtime.file_statistics_cache_limit = '10M'
+```
+
+To disable the file statistics cache, set the limit to 0.
+
+The file statistics cache is no longer created inside the `ListingTable`.
+Instead, it is created within the `CacheManager` and must be passed to the
`ListingTable`.
+
+**Who is affected:**
+
+- Users who want to limit the memory usage of the file statistics cache.
+- Users who want to disable the file statistics cache.
+- Users who want to create a `ListingTable` programmatically with a file
statistics cache.
+
+**Migration guide:**
+
+Disable the cache by setting the configuration value to 0:
+
+```sql
+SET datafusion.runtime.file_statistics_cache_limit = '0'
+```
+
+Use the file statistics cache provided by the `CacheManager` when initializing
a new `ListingTable`:
+
+```rust,ignore
+ListingTable::try_new(config)?
+ .with_cache(ctx.runtime_env().cache_manager.get_file_statistic_cache())
+```
diff --git a/_sources/user-guide/configs.md.txt
b/_sources/user-guide/configs.md.txt
index f7f9426e2b..f0bab1c4db 100644
--- a/_sources/user-guide/configs.md.txt
+++ b/_sources/user-guide/configs.md.txt
@@ -229,14 +229,15 @@ SET datafusion.runtime.memory_limit = '2G';
The following runtime configuration settings are available:
-| key | default | description
|
-| ------------------------------------------ | ------- |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
-| datafusion.runtime.list_files_cache_limit | 1M | Maximum memory to use
for list files cache. Supports suffixes K (kilobytes), M (megabytes), and G
(gigabytes) or '0' for 0. Example: '2G' for 2 gigabytes.
|
-| datafusion.runtime.list_files_cache_ttl | NULL | TTL (time-to-live) of
the entries in the list file cache. Supports units m (minutes), and s
(seconds). Example: '2m' for 2 minutes.
|
-| datafusion.runtime.max_temp_directory_size | 100G | Maximum temporary
file directory size. Supports suffixes K (kilobytes), M (megabytes), and G
(gigabytes) or '0' for 0. Example: '2G' for 2 gigabytes.
|
-| datafusion.runtime.memory_limit | NULL | Maximum memory limit
for query execution. Supports suffixes K (kilobytes), M (megabytes), and G
(gigabytes) or '0' for 0. Example: '2G' for 2 gigabytes.
|
-| datafusion.runtime.metadata_cache_limit | 50M | Maximum memory to use
for file metadata cache such as Parquet metadata. Supports suffixes K
(kilobytes), M (megabytes), and G (gigabytes) or '0' for 0. Example: '2G' for 2
gigabytes. |
-| datafusion.runtime.temp_directory | NULL | The path to the
temporary file directory.
|
+| key | default | description
|
+| ---------------------------------------------- | ------- |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
+| datafusion.runtime.file_statistics_cache_limit | 20M | Maximum memory to
use for file statistics cache. Supports suffixes K (kilobytes), M (megabytes),
and G (gigabytes) or '0' for 0. Example: '2G' for 2 gigabytes.
|
+| datafusion.runtime.list_files_cache_limit | 1M | Maximum memory to
use for list files cache. Supports suffixes K (kilobytes), M (megabytes), and G
(gigabytes) or '0' for 0. Example: '2G' for 2 gigabytes.
|
+| datafusion.runtime.list_files_cache_ttl | NULL | TTL
(time-to-live) of the entries in the list file cache. Supports units m
(minutes), and s (seconds). Example: '2m' for 2 minutes.
|
+| datafusion.runtime.max_temp_directory_size | 100G | Maximum temporary
file directory size. Supports suffixes K (kilobytes), M (megabytes), and G
(gigabytes) or '0' for 0. Example: '2G' for 2 gigabytes.
|
+| datafusion.runtime.memory_limit | NULL | Maximum memory
limit for query execution. Supports suffixes K (kilobytes), M (megabytes), and
G (gigabytes) or '0' for 0. Example: '2G' for 2 gigabytes.
|
+| datafusion.runtime.metadata_cache_limit | 50M | Maximum memory to
use for file metadata cache such as Parquet metadata. Supports suffixes K
(kilobytes), M (megabytes), and G (gigabytes) or '0' for 0. Example: '2G' for 2
gigabytes. |
+| datafusion.runtime.temp_directory | NULL | The path to the
temporary file directory.
|
# Tuning Guide
diff --git a/library-user-guide/upgrading/54.0.0.html
b/library-user-guide/upgrading/54.0.0.html
index bb66d684f8..17ba8d1822 100644
--- a/library-user-guide/upgrading/54.0.0.html
+++ b/library-user-guide/upgrading/54.0.0.html
@@ -1084,6 +1084,34 @@ move the borrow behind an interior-mutability primitive
such as <code class="doc
or <code class="docutils literal notranslate"><span
class="pre">Arc<OnceLock<_>></span></code>.</p>
<p>See <a class="reference external"
href="https://github.com/apache/datafusion/pull/21803";>PR #21803</a> for
details.</p>
</section>
+<section
id="file-statistics-cache-is-now-memory-limited-and-managed-by-the-cachemanager">
+<h3>File statistics cache is now memory-limited and managed by the <code
class="docutils literal notranslate"><span
class="pre">CacheManager</span></code><a class="headerlink"
href="#file-statistics-cache-is-now-memory-limited-and-managed-by-the-cachemanager"
title="Link to this heading">#</a></h3>
+<p>The file statistics cache used by <code class="docutils literal
notranslate"><span class="pre">ListingTable</span></code> is now memory-limited
and
+centrally managed through the <code class="docutils literal notranslate"><span
class="pre">CacheManager</span></code>.</p>
+<p>To configure the cache size use the <code class="docutils literal
notranslate"><span class="pre">file_statistics_cache_limit</span></code>
setting:</p>
+<div class="highlight-sql notranslate"><div
class="highlight"><pre><span></span><span class="k">SET</span><span class="w">
</span><span class="n">datafusion</span><span class="p">.</span><span
class="n">runtime</span><span class="p">.</span><span
class="n">file_statistics_cache_limit</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span class="s1">'10M'</span>
+</pre></div>
+</div>
+<p>To disable the file statistics cache, set the limit to 0.</p>
+<p>The file statistics cache is no longer created inside the <code
class="docutils literal notranslate"><span
class="pre">ListingTable</span></code>.
+Instead, it is created within the <code class="docutils literal
notranslate"><span class="pre">CacheManager</span></code> and must be passed to
the <code class="docutils literal notranslate"><span
class="pre">ListingTable</span></code>.</p>
+<p><strong>Who is affected:</strong></p>
+<ul class="simple">
+<li><p>Users who want to limit the memory usage of the file statistics
cache.</p></li>
+<li><p>Users who want to disable the file statistics cache.</p></li>
+<li><p>Users who want to create a <code class="docutils literal
notranslate"><span class="pre">ListingTable</span></code> programmatically with
a file statistics cache.</p></li>
+</ul>
+<p><strong>Migration guide:</strong></p>
+<p>Disable the cache by setting the configuration value to 0:</p>
+<div class="highlight-sql notranslate"><div
class="highlight"><pre><span></span><span class="k">SET</span><span class="w">
</span><span class="n">datafusion</span><span class="p">.</span><span
class="n">runtime</span><span class="p">.</span><span
class="n">file_statistics_cache_limit</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span class="s1">'0'</span>
+</pre></div>
+</div>
+<p>Use the file statistics cache provided by the <code class="docutils literal
notranslate"><span class="pre">CacheManager</span></code> when initializing a
new <code class="docutils literal notranslate"><span
class="pre">ListingTable</span></code>:</p>
+<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="n">ListingTable</span><span
class="p">::</span><span class="n">try_new</span><span class="p">(</span><span
class="n">config</span><span class="p">)</span><span class="o">?</span>
+<span class="w"> </span><span class="p">.</span><span
class="n">with_cache</span><span class="p">(</span><span
class="n">ctx</span><span class="p">.</span><span
class="n">runtime_env</span><span class="p">().</span><span
class="n">cache_manager</span><span class="p">.</span><span
class="n">get_file_statistic_cache</span><span class="p">())</span>
+</pre></div>
+</div>
+</section>
</section>
</section>
@@ -1161,6 +1189,7 @@ or <code class="docutils literal notranslate"><span
class="pre">Arc<OnceLock&
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#arrays-zip-struct-field-names-changed"><code class="docutils literal
notranslate"><span class="pre">arrays_zip</span></code> struct field names
changed</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#box-c-and-arc-c-treenodecontainer-impls-now-require-c-default"><code
class="docutils literal notranslate"><span
class="pre">Box<C></span></code> and <code class="docutils literal
notranslate"><span class="pre">Arc<C></span></code> <code class="docutils
literal notranslate"><span class="pre">TreeNodeContainer</span></code> impls
now require <code class="docutils literal notranslate"><span class [...]
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#memorypool-now-requires-static-adds-any-as-a-supertrait"><code
class="docutils literal notranslate"><span class="pre">MemoryPool</span></code>
now requires <code class="docutils literal notranslate"><span
class="pre">'static</span></code> (adds <code class="docutils literal
notranslate"><span class="pre">Any</span></code> as a supertrait)</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#file-statistics-cache-is-now-memory-limited-and-managed-by-the-cachemanager">File
statistics cache is now memory-limited and managed by the <code
class="docutils literal notranslate"><span
class="pre">CacheManager</span></code></a></li>
</ul>
</li>
</ul>
diff --git a/searchindex.js b/searchindex.js
index f072fd6903..14467b528b 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"!=":[[73,"op-neq"]],"!~":[[73,"op-re-not-match"]],"!~*":[[73,"op-re-not-match-i"]],"!~~":[[73,"id19"]],"!~~*":[[73,"id20"]],"#":[[73,"op-bit-xor"]],"%":[[73,"op-modulo"]],"&":[[73,"op-bit-and"]],"(relation,
name) tuples in logical fields and logical columns are
unique":[[15,"relation-name-tuples-in-logical-fields-and-logical-columns-are-unique"]],"*":[[73,"op-multiply"]],"+":[[73,"op-plus"]],"-":[[73,"op-minus"]],"/":[[73,"op-divide"]],"1.
Array Literal Con [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"!=":[[73,"op-neq"]],"!~":[[73,"op-re-not-match"]],"!~*":[[73,"op-re-not-match-i"]],"!~~":[[73,"id19"]],"!~~*":[[73,"id20"]],"#":[[73,"op-bit-xor"]],"%":[[73,"op-modulo"]],"&":[[73,"op-bit-and"]],"(relation,
name) tuples in logical fields and logical columns are
unique":[[15,"relation-name-tuples-in-logical-fields-and-logical-columns-are-unique"]],"*":[[73,"op-multiply"]],"+":[[73,"op-plus"]],"-":[[73,"op-minus"]],"/":[[73,"op-divide"]],"1.
Array Literal Con [...]
\ No newline at end of file
diff --git a/user-guide/configs.html b/user-guide/configs.html
index efaf6c73f7..f7b432a0ea 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -1049,27 +1049,31 @@ example, to set and reset <code class="docutils literal
notranslate"><span class
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p>datafusion.runtime.list_files_cache_limit</p></td>
+<tr
class="row-even"><td><p>datafusion.runtime.file_statistics_cache_limit</p></td>
+<td><p>20M</p></td>
+<td><p>Maximum memory to use for file statistics cache. Supports suffixes K
(kilobytes), M (megabytes), and G (gigabytes) or ‘0’ for 0. Example: ‘2G’ for 2
gigabytes.</p></td>
+</tr>
+<tr class="row-odd"><td><p>datafusion.runtime.list_files_cache_limit</p></td>
<td><p>1M</p></td>
<td><p>Maximum memory to use for list files cache. Supports suffixes K
(kilobytes), M (megabytes), and G (gigabytes) or ‘0’ for 0. Example: ‘2G’ for 2
gigabytes.</p></td>
</tr>
-<tr class="row-odd"><td><p>datafusion.runtime.list_files_cache_ttl</p></td>
+<tr class="row-even"><td><p>datafusion.runtime.list_files_cache_ttl</p></td>
<td><p>NULL</p></td>
<td><p>TTL (time-to-live) of the entries in the list file cache. Supports
units m (minutes), and s (seconds). Example: ‘2m’ for 2 minutes.</p></td>
</tr>
-<tr class="row-even"><td><p>datafusion.runtime.max_temp_directory_size</p></td>
+<tr class="row-odd"><td><p>datafusion.runtime.max_temp_directory_size</p></td>
<td><p>100G</p></td>
<td><p>Maximum temporary file directory size. Supports suffixes K (kilobytes),
M (megabytes), and G (gigabytes) or ‘0’ for 0. Example: ‘2G’ for 2
gigabytes.</p></td>
</tr>
-<tr class="row-odd"><td><p>datafusion.runtime.memory_limit</p></td>
+<tr class="row-even"><td><p>datafusion.runtime.memory_limit</p></td>
<td><p>NULL</p></td>
<td><p>Maximum memory limit for query execution. Supports suffixes K
(kilobytes), M (megabytes), and G (gigabytes) or ‘0’ for 0. Example: ‘2G’ for 2
gigabytes.</p></td>
</tr>
-<tr class="row-even"><td><p>datafusion.runtime.metadata_cache_limit</p></td>
+<tr class="row-odd"><td><p>datafusion.runtime.metadata_cache_limit</p></td>
<td><p>50M</p></td>
<td><p>Maximum memory to use for file metadata cache such as Parquet metadata.
Supports suffixes K (kilobytes), M (megabytes), and G (gigabytes) or ‘0’ for 0.
Example: ‘2G’ for 2 gigabytes.</p></td>
</tr>
-<tr class="row-odd"><td><p>datafusion.runtime.temp_directory</p></td>
+<tr class="row-even"><td><p>datafusion.runtime.temp_directory</p></td>
<td><p>NULL</p></td>
<td><p>The path to the temporary file directory.</p></td>
</tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
