This is an automated email from the ASF dual-hosted git repository.
HeartSaVioR pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new 34c536097e16 [SPARK-58925][SS][DOC] Document maxOpenFiles
troubleshooting guidance
34c536097e16 is described below
commit 34c536097e16e075ca4e7c9bb9616546a84bc64f
Author: Livia Zhu <[email protected]>
AuthorDate: Fri Aug 21 12:29:06 2026 +0900
[SPARK-58925][SS][DOC] Document maxOpenFiles troubleshooting guidance
### What changes were proposed in this pull request?
Document that users should set
`spark.sql.streaming.stateStore.rocksdb.maxOpenFiles` to a positive value
when a
streaming query fails to open RocksDB with `Too many open files`.
JIRA: https://issues.apache.org/jira/browse/SPARK-58925
### Why are the changes needed?
The existing documentation explains the open-file cache behavior but does
not
tell users how to configure RocksDB after reaching the operating system's
open
file limit.
### Does this PR introduce _any_ user-facing change?
No. This is a documentation-only update.
### How was this patch tested?
Validated with `git diff --check`. No runtime tests are applicable to this
documentation-only change.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
Closes #58183 from liviazhu/SPARK-58925-rocksdb-max-open-files-doc.
Authored-by: Livia Zhu <[email protected]>
Signed-off-by: Jungtaek Lim <[email protected]>
(cherry picked from commit ebdf5543b970c5c563037795ea66e84e44ad9588)
Signed-off-by: Jungtaek Lim <[email protected]>
---
docs/streaming/apis-on-dataframes-and-datasets.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/streaming/apis-on-dataframes-and-datasets.md
b/docs/streaming/apis-on-dataframes-and-datasets.md
index 86585caead51..e946a8e38b89 100644
--- a/docs/streaming/apis-on-dataframes-and-datasets.md
+++ b/docs/streaming/apis-on-dataframes-and-datasets.md
@@ -1851,7 +1851,7 @@ Here are the configs regarding to RocksDB instance of the
state store provider:
</tr>
<tr>
<td>spark.sql.streaming.stateStore.rocksdb.maxOpenFiles</td>
- <td>The number of open files that can be used by the RocksDB instance.
Value of -1 means that files opened are always kept open. If the open file
limit is reached, RocksDB will evict entries from the open file cache and close
those file descriptors and remove the entries from the cache.</td>
+ <td>The number of open files that can be used by the RocksDB instance.
Value of -1 means that files opened are always kept open. If the open file
limit is reached, RocksDB will evict entries from the open file cache and close
those file descriptors and remove the entries from the cache. Set this
configuration to a positive value if a query fails when opening RocksDB with
<code>Too many open files</code>.</td>
<td>-1</td>
</tr>
<tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
