This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 27a7819a3e Publish built docs triggered by
8166586fa767008a45665358d200ce23ec955f64
27a7819a3e is described below
commit 27a7819a3e50465b8f48a2c7c7f94d4d61f210fe
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Thu Sep 24 10:41:20 2026 +0000
Publish built docs triggered by 8166586fa767008a45665358d200ce23ec955f64
---
...id-013012468fa27c9e9fdb0a14196d357c619c68a8.svg | 2 +-
_sources/about/versioning_policy.md.txt | 73 +++++++++++++++++++++-
.../contributor-guide/config_conventions.md.txt | 48 +++++++++++++-
_sources/contributor-guide/development.md.txt | 6 ++
_sources/user-guide/latest/configs.md.txt | 13 ++++
_sources/user-guide/latest/tuning.md.txt | 8 +++
about/versioning_policy.html | 72 +++++++++++++++++++--
contributor-guide/config_conventions.html | 65 ++++++++++++++++++-
contributor-guide/development.html | 5 ++
searchindex.js | 2 +-
user-guide/latest/configs.html | 11 ++++
user-guide/latest/tuning.html | 7 +++
12 files changed, 299 insertions(+), 13 deletions(-)
diff --git a/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
index 5342794c8d..ede7c29a72 100644
--- a/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
+++ b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg";
xmlns:xlink="http://www.w3.org/1999/xlink"; class="flowchart" style="max-width:
511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 942"
role="graphics-document document"
aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet
ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes
edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes
dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg";
xmlns:xlink="http://www.w3.org/1999/xlink"; class="flowchart" style="max-width:
511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 942"
role="graphics-document document"
aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet
ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes
edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes
dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/about/versioning_policy.md.txt
b/_sources/about/versioning_policy.md.txt
index e5cb832f44..7b6e4974c8 100644
--- a/_sources/about/versioning_policy.md.txt
+++ b/_sources/about/versioning_policy.md.txt
@@ -48,7 +48,14 @@ interface that vendors implement. It is small on purpose,
and everything outside
The following are covered by this versioning policy:
- **Configuration keys under `spark.comet.*`**: their names, types, accepted
values, default
- values, and semantics.
+ values, and semantics. Two classes of key are excluded — those in the
`testing` category, and
+ those marked internal; see
+ [Testing and Internal Configurations Are
Exempt](#testing-and-internal-configurations-are-exempt).
+ Every other key is covered wherever Comet documents it, including keys the
+ [configuration reference](../user-guide/latest/configs.md) does not list in
its own right: the
+ per-expression `spark.comet.expression.<Name>.allowIncompatible` opt-ins,
which the
+ [compatibility guide](../user-guide/latest/compatibility/index.md)
documents, and the deprecated
+ alias a rename leaves behind.
- **A small, enumerated public Java and Scala API**: the class names users
write into Spark config
properties, and the S3 credential provider SPI that vendors implement. The
full list is in
[Public Scala and Java API](#public-scala-and-java-api).
@@ -59,6 +66,9 @@ The following are covered by this versioning policy:
The following are internal implementation details. They are not covered by
this policy and may
change in any release:
+- Configuration keys in the `testing` category, and keys marked internal. They
exist to let Comet's
+ own suites and maintainers reach internal behavior, and they expose that
behavior directly. See
+ [Testing and Internal Configurations Are
Exempt](#testing-and-internal-configurations-are-exempt).
- The protobuf format used to serialize query plans between the JVM and the
native library. The
JVM jar and the native library ship together and are versioned together; see
[Native Library Coupling](#native-library-coupling).
@@ -71,6 +81,52 @@ change in any release:
fall back to Spark. An expression that ran natively in one release may fall
back in the next,
and vice versa. The results stay the same; only the speed changes.
+### Testing and Internal Configurations Are Exempt
+
+Two kinds of configuration key sit outside this policy, and what puts a key
outside is an explicit
+mark on its declaration in `CometConf.scala`: **a key is exempt only if it is
in the `testing`
+category or is marked `internal()`.** Every other `spark.comet.*` key is
covered.
+
+Absence from the [configuration reference](../user-guide/latest/configs.md) is
not the test,
+because that page is not an exhaustive list of covered keys. The per-expression
+`spark.comet.expression.<Name>.allowIncompatible` opt-ins are read as plain
strings with no
+`ConfigEntry` behind them, so no generated table ever lists them; the
+[compatibility guide](../user-guide/latest/compatibility/index.md) is where
they are documented
+instead. A deprecated alias left behind by a rename has no row of its own
either. Both are fully
+covered.
+
+The two mechanisms that do exempt a key are set independently of each other:
+
+- **The `testing` category.** Every key declares a category, and `testing`
routes it into the
+ Development & Testing Settings table rather than in with the production
settings. These keys
+ exist so that Comet's own suites, and contributors chasing a bug, can reach
a state the rest of
+ the code is not built to support: disabling native scans to isolate a
planner problem, running
+ Comet in on-heap mode, making a declined operator throw instead of quietly
reporting itself, or
+ running a partial aggregate without its final counterpart. Setting one is a
debugging step, not a
+ deployment choice.
+- **`internal()`.** A key marked internal is left out of the configuration
reference entirely, so
+ Comet never publishes its name, its default, or what it does. A user has no
supported way to
+ learn that it exists, which is the point: these are escape hatches a
maintainer reaches for while
+ working on Comet itself.
+
+For a key caught by either mechanism, the name, type, accepted values, default
value, and semantics
+may change in any release, including a patch release, and the key may be
removed outright. None of
+the machinery the rest of this policy requires applies: no
`spark.comet.legacy.*` escape hatch for a
+behavior change, no deprecated alias for a rename, no deprecation cycle before
removal, and no
+upgrade guide entry.
+
+The exemption exists because these keys point at Comet's internals by
construction. Guaranteeing
+them across releases would pin the implementation details they expose, which
is the thing the rest
+of this section deliberately leaves free to change.
+
+The corollary binds contributors: **neither mechanism may be the only way to
reach a behavior that
+production users need.** If a knob turns out to be one that deployments
legitimately set, it belongs
+in a non-`testing` category and must not be marked internal, and the
guarantees come with it.
+Choosing a category, and deciding whether to mark a key internal, are
therefore policy decisions
+rather than routing details; see
+[Categories and
Visibility](../contributor-guide/config_conventions.md#categories-and-visibility)
+in the contributor guide.
+
## What Each Version Component Means
### Major Releases
@@ -99,8 +155,11 @@ A minor release may:
### Patch Releases
A patch release contains bug fixes only. It adds no configuration keys and
makes no behavior
-changes, with one exception: correctness fixes, which are covered in
-[Correctness Fixes Are Not Breaking
Changes](#correctness-fixes-are-not-breaking-changes).
+changes, with two exceptions: correctness fixes, which are covered in
+[Correctness Fixes Are Not Breaking
Changes](#correctness-fixes-are-not-breaking-changes), and
+configuration keys in the `testing` category or marked internal, which are
outside the policy
+altogether and may be added, changed, or removed in any release. See
+[Testing and Internal Configurations Are
Exempt](#testing-and-internal-configurations-are-exempt).
## Behavior Changes and Legacy Configurations
@@ -125,6 +184,10 @@ Behavior changes that require this treatment include
changing the default value
configuration key, changing what an existing key's values mean, and changing
the semantics of an
`Incompatible` expression or operator whose divergence from Spark users may
have come to depend on.
+Changing a `testing` or internal key is not a behavior change for this
purpose, and needs no escape
+hatch. See
+[Testing and Internal Configurations Are
Exempt](#testing-and-internal-configurations-are-exempt).
+
### Lifetime of a Legacy Configuration
A `spark.comet.legacy.*` key is deprecated from the moment it is added. Its
purpose is to buy users
@@ -147,6 +210,10 @@ The alias may only be dropped in a major release.
Removing a configuration key outright requires a deprecation cycle: the key
must remain available,
with a deprecation warning, for at least one minor release before it is
removed in a major release.
+Neither rule applies to a `testing` or internal key, which may be renamed
without an alias and
+removed in any release. See
+[Testing and Internal Configurations Are
Exempt](#testing-and-internal-configurations-are-exempt).
+
## Correctness Fixes Are Not Breaking Changes
Comet's contract is to produce the results that Apache Spark produces. When an
expression or
diff --git a/_sources/contributor-guide/config_conventions.md.txt
b/_sources/contributor-guide/config_conventions.md.txt
index 5bcad087ea..6c198f740c 100644
--- a/_sources/contributor-guide/config_conventions.md.txt
+++ b/_sources/contributor-guide/config_conventions.md.txt
@@ -66,19 +66,60 @@ The symbol name is what appears in code; the key is what
appears in user configu
The two do not have to match segment-for-segment — brevity in the symbol is
fine as long
as the key remains descriptive.
-## Categories
+## Categories and Visibility
Every `ConfigEntry` must call `.category(...)`. The category is used to route
the key into
the right table in the user guide's `configs.md`. Available categories are
declared as
`CATEGORY_*` constants at the top of `CometConf.scala`. If a new config does
not fit an
existing category, discuss adding a new one before landing the config.
+A second, independent choice is whether to call `.internal()`, which keeps the
key out of
+`configs.md` altogether.
+
+Neither choice is cosmetic. Together they decide whether Comet's
+[versioning policy](../about/versioning_policy.md) covers the key. The
policy's rule keys on these
+two marks and nothing else: a key is exempt only if it is in the `testing`
category or is marked
+`internal()`, and every other `spark.comet.*` key is covered.
+
+| Category | `internal()` | Appears in `configs.md` | Covered by the
versioning policy |
+| --------- | ------------ | -------------------------------- |
-------------------------------- |
+| any other | no | yes, in its category's table | **yes**
|
+| `testing` | no | yes, under Development & Testing | no
|
+| any | yes | no | no
|
+
+Note that a key read by string rather than through a registered `ConfigEntry`
— as the
+per-expression `spark.comet.expression.<Name>.allowIncompatible` opt-ins are —
has no way to carry
+either mark, so it is covered. Reaching for a dynamic key is therefore not a
way to avoid the
+guarantee; if the key is a debugging aid, give it a real `ConfigEntry` in the
`testing` category.
+
+Being covered commits the project to the key's name, type, accepted values,
default, and semantics
+across minor releases. Being
+[exempt](../about/versioning_policy.md#testing-and-internal-configurations-are-exempt)
means the key
+may be renamed, retyped, redefaulted, or removed in any release, including a
patch release, with no
+alias, no deprecation cycle, and no upgrade guide entry.
+
+So reach for `testing` or `internal()` only when the key exists to let Comet's
own suites, or
+someone debugging Comet itself, reach a state the rest of the code is not
built to support —
+disabling native scans, running in on-heap mode, making a declined operator
throw. Use `internal()`
+in particular when Comet should not publish the key at all, so that nobody can
adopt it in the first
+place.
+
+If a deployment would have a legitimate reason to set the key, it belongs in a
non-`testing`
+category and must not be `internal()`, and the guarantees come with it. Do not
use either mechanism
+as a way to ship a production knob without committing to it.
+
## Renaming an Existing Config
Configs under `spark.comet.*` are stable across minor releases: users may have
set them in
production `spark-defaults.conf` files, Spark job submissions, or notebooks.
Renaming a key
must not silently break those deployments.
+Keys in the `testing` category, and keys marked `internal()`, are
+[exempt from the versioning
policy](../about/versioning_policy.md#testing-and-internal-configurations-are-exempt)
+and may be renamed outright: skip the `withAlternative` call in step 1, and
step 5 with it. The
+rest of the checklist still applies, because a stale key string left behind in
code or docs is a
+bug either way.
+
Use the `withAlternative` builder on `ConfigBuilder` to keep the old key
working as a
deprecated alias:
@@ -159,5 +200,10 @@ Two cases do **not** need a legacy config:
- **Changes to which expressions and operators run natively.** Falling back to
Spark, or ceasing
to, changes performance rather than results.
+Nor does a change to a key in the `testing` category or a key marked
`internal()`, both
+[exempt from the versioning
policy](../about/versioning_policy.md#testing-and-internal-configurations-are-exempt):
+the default and the meaning may change in any release, with no legacy config
and no upgrade guide
+entry. Update the suites that set it in the same PR.
+
Removing a legacy config is a major-release change, handled the same way as
removing a deprecated
alias.
diff --git a/_sources/contributor-guide/development.md.txt
b/_sources/contributor-guide/development.md.txt
index b5752e0863..2bf3bd6ec8 100644
--- a/_sources/contributor-guide/development.md.txt
+++ b/_sources/contributor-guide/development.md.txt
@@ -567,6 +567,12 @@ It is possible to debug both native and JVM code
concurrently as described in th
## Submitting a Pull Request
+Use `git push` for normal updates to your PR branch. If you need to force push
after a rebase
+or amend, use `git push --force-with-lease` instead of `git push --force` (or
`-f`). This reduces
+the risk of accidentally overwriting another maintainer's commits when
multiple people push
+to the same PR branch. If the lease check rejects the push, inspect and
integrate the remote
+changes before retrying; do not switch to `--force` to bypass the check.
+
Before submitting a pull request, follow this checklist to ensure your changes
are ready:
### 1. Format Your Code
diff --git a/_sources/user-guide/latest/configs.md.txt
b/_sources/user-guide/latest/configs.md.txt
index 0290e84751..9793555e09 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -160,6 +160,19 @@ shuffle, including applications that rely only on remote
reliable storage or dec
## Development & Testing Settings
+These settings exist for Comet's own test suites and for debugging. They are
**not covered by the
+[versioning
policy](../../about/versioning_policy.md#testing-and-internal-configurations-are-exempt)**:
+their names, defaults, accepted values, and meanings may change in any
release, including a patch
+release, and any of them may be removed without a deprecation cycle. Do not
set them in
+production.
+
+Comet also marks a handful of keys internal and deliberately leaves them off
this page entirely.
+They are maintainer escape hatches, not settings, and carry no guarantee of
any kind. Absence from
+this page does not by itself mean that, though: the per-expression
+`spark.comet.expression.<Name>.allowIncompatible` opt-ins are documented in the
+[compatibility guide](compatibility/index.md) rather than here, and the
versioning policy covers
+them like any other production setting.
+
<!--BEGIN:CONFIG_TABLE[testing]-->
<!-- prettier-ignore-start -->
| Config | Description | Default Value |
diff --git a/_sources/user-guide/latest/tuning.md.txt
b/_sources/user-guide/latest/tuning.md.txt
index 368f46ece6..5855d6cdc0 100644
--- a/_sources/user-guide/latest/tuning.md.txt
+++ b/_sources/user-guide/latest/tuning.md.txt
@@ -381,6 +381,14 @@ projects the file schema. The original null checks and
residual runtime filter r
The original join still verifies matches, including any hash collisions
admitted by the filter.
Standalone projections, other filter expressions, and limits prevent reader
attachment.
+To preserve schema-conversion and timestamp-overflow errors, runtime reader
pruning is disabled for
+each file whose projected or statically filtered columns require schema
adaptations beyond direct
+column mappings or literal values. This conservative check also disables
reader pruning for allowed
+`INT32` to `BIGINT` promotion and for projecting a subset of a struct's
fields, even when those
+adaptations cannot fail. Nested column pruning still reads only the requested
struct fields. Scans
+with supplied file statistics also skip reader attachment. These cases still
use runtime filtering
+on decoded batches.
+
Filters stay within the task's native plan and do not propagate across Spark
exchanges or JVM/Arrow
boundaries. A shuffled hash join can still filter probe batches after shuffle,
but it cannot send
its filter back to an earlier scan stage. Compare the [runtime-filter and scan
metrics](metrics.md#hash-joins)
diff --git a/about/versioning_policy.html b/about/versioning_policy.html
index de7cfb03ea..ee407934ad 100644
--- a/about/versioning_policy.html
+++ b/about/versioning_policy.html
@@ -503,7 +503,14 @@ interface that vendors implement. It is small on purpose,
and everything outside
<p>The following are covered by this versioning policy:</p>
<ul class="simple">
<li><p><strong>Configuration keys under <code class="docutils literal
notranslate"><span class="pre">spark.comet.*</span></code></strong>: their
names, types, accepted values, default
-values, and semantics.</p></li>
+values, and semantics. Two classes of key are excluded — those in the <code
class="docutils literal notranslate"><span class="pre">testing</span></code>
category, and
+those marked internal; see
+<a class="reference internal"
href="#testing-and-internal-configurations-are-exempt">Testing and Internal
Configurations Are Exempt</a>.
+Every other key is covered wherever Comet documents it, including keys the
+<a class="reference internal" href="../user-guide/latest/configs.html"><span
class="std std-doc">configuration reference</span></a> does not list in its own
right: the
+per-expression <code class="docutils literal notranslate"><span
class="pre">spark.comet.expression.<Name>.allowIncompatible</span></code>
opt-ins, which the
+<a class="reference internal"
href="../user-guide/latest/compatibility/index.html"><span class="std
std-doc">compatibility guide</span></a> documents, and the deprecated
+alias a rename leaves behind.</p></li>
<li><p><strong>A small, enumerated public Java and Scala API</strong>: the
class names users write into Spark config
properties, and the S3 credential provider SPI that vendors implement. The
full list is in
<a class="reference internal" href="#public-scala-and-java-api">Public Scala
and Java API</a>.</p></li>
@@ -514,6 +521,9 @@ contract is defined by Apache Spark rather than by Comet.
See
<p>The following are internal implementation details. They are not covered by
this policy and may
change in any release:</p>
<ul class="simple">
+<li><p>Configuration keys in the <code class="docutils literal
notranslate"><span class="pre">testing</span></code> category, and keys marked
internal. They exist to let Comet’s
+own suites and maintainers reach internal behavior, and they expose that
behavior directly. See
+<a class="reference internal"
href="#testing-and-internal-configurations-are-exempt">Testing and Internal
Configurations Are Exempt</a>.</p></li>
<li><p>The protobuf format used to serialize query plans between the JVM and
the native library. The
JVM jar and the native library ship together and are versioned together; see
<a class="reference internal" href="#native-library-coupling">Native Library
Coupling</a>.</p></li>
@@ -526,6 +536,48 @@ native block is partitioned, and how a plan is rendered
may all change between r
fall back to Spark. An expression that ran natively in one release may fall
back in the next,
and vice versa. The results stay the same; only the speed changes.</p></li>
</ul>
+<section id="testing-and-internal-configurations-are-exempt">
+<h3>Testing and Internal Configurations Are Exempt<a class="headerlink"
href="#testing-and-internal-configurations-are-exempt" title="Link to this
heading">#</a></h3>
+<p>Two kinds of configuration key sit outside this policy, and what puts a key
outside is an explicit
+mark on its declaration in <code class="docutils literal notranslate"><span
class="pre">CometConf.scala</span></code>: <strong>a key is exempt only if it
is in the <code class="docutils literal notranslate"><span
class="pre">testing</span></code>
+category or is marked <code class="docutils literal notranslate"><span
class="pre">internal()</span></code>.</strong> Every other <code
class="docutils literal notranslate"><span
class="pre">spark.comet.*</span></code> key is covered.</p>
+<p>Absence from the <a class="reference internal"
href="../user-guide/latest/configs.html"><span class="std
std-doc">configuration reference</span></a> is not the test,
+because that page is not an exhaustive list of covered keys. The per-expression
+<code class="docutils literal notranslate"><span
class="pre">spark.comet.expression.<Name>.allowIncompatible</span></code>
opt-ins are read as plain strings with no
+<code class="docutils literal notranslate"><span
class="pre">ConfigEntry</span></code> behind them, so no generated table ever
lists them; the
+<a class="reference internal"
href="../user-guide/latest/compatibility/index.html"><span class="std
std-doc">compatibility guide</span></a> is where they are documented
+instead. A deprecated alias left behind by a rename has no row of its own
either. Both are fully
+covered.</p>
+<p>The two mechanisms that do exempt a key are set independently of each
other:</p>
+<ul class="simple">
+<li><p><strong>The <code class="docutils literal notranslate"><span
class="pre">testing</span></code> category.</strong> Every key declares a
category, and <code class="docutils literal notranslate"><span
class="pre">testing</span></code> routes it into the
+Development & Testing Settings table rather than in with the production
settings. These keys
+exist so that Comet’s own suites, and contributors chasing a bug, can reach a
state the rest of
+the code is not built to support: disabling native scans to isolate a planner
problem, running
+Comet in on-heap mode, making a declined operator throw instead of quietly
reporting itself, or
+running a partial aggregate without its final counterpart. Setting one is a
debugging step, not a
+deployment choice.</p></li>
+<li><p><strong><code class="docutils literal notranslate"><span
class="pre">internal()</span></code>.</strong> A key marked internal is left
out of the configuration reference entirely, so
+Comet never publishes its name, its default, or what it does. A user has no
supported way to
+learn that it exists, which is the point: these are escape hatches a
maintainer reaches for while
+working on Comet itself.</p></li>
+</ul>
+<p>For a key caught by either mechanism, the name, type, accepted values,
default value, and semantics
+may change in any release, including a patch release, and the key may be
removed outright. None of
+the machinery the rest of this policy requires applies: no <code
class="docutils literal notranslate"><span
class="pre">spark.comet.legacy.*</span></code> escape hatch for a
+behavior change, no deprecated alias for a rename, no deprecation cycle before
removal, and no
+upgrade guide entry.</p>
+<p>The exemption exists because these keys point at Comet’s internals by
construction. Guaranteeing
+them across releases would pin the implementation details they expose, which
is the thing the rest
+of this section deliberately leaves free to change.</p>
+<p>The corollary binds contributors: <strong>neither mechanism may be the only
way to reach a behavior that
+production users need.</strong> If a knob turns out to be one that deployments
legitimately set, it belongs
+in a non-<code class="docutils literal notranslate"><span
class="pre">testing</span></code> category and must not be marked internal, and
the guarantees come with it.
+Choosing a category, and deciding whether to mark a key internal, are
therefore policy decisions
+rather than routing details; see
+<a class="reference internal"
href="../contributor-guide/config_conventions.html#categories-and-visibility"><span
class="std std-ref">Categories and Visibility</span></a>
+in the contributor guide.</p>
+</section>
</section>
<section id="what-each-version-component-means">
<h2>What Each Version Component Means<a class="headerlink"
href="#what-each-version-component-means" title="Link to this
heading">#</a></h2>
@@ -557,8 +609,11 @@ public API that keep existing vendor jars working.</p></li>
<section id="patch-releases">
<h3>Patch Releases<a class="headerlink" href="#patch-releases" title="Link to
this heading">#</a></h3>
<p>A patch release contains bug fixes only. It adds no configuration keys and
makes no behavior
-changes, with one exception: correctness fixes, which are covered in
-<a class="reference internal"
href="#correctness-fixes-are-not-breaking-changes">Correctness Fixes Are Not
Breaking Changes</a>.</p>
+changes, with two exceptions: correctness fixes, which are covered in
+<a class="reference internal"
href="#correctness-fixes-are-not-breaking-changes">Correctness Fixes Are Not
Breaking Changes</a>, and
+configuration keys in the <code class="docutils literal notranslate"><span
class="pre">testing</span></code> category or marked internal, which are
outside the policy
+altogether and may be added, changed, or removed in any release. See
+<a class="reference internal"
href="#testing-and-internal-configurations-are-exempt">Testing and Internal
Configurations Are Exempt</a>.</p>
</section>
</section>
<section id="behavior-changes-and-legacy-configurations">
@@ -581,6 +636,9 @@ pin an old Comet release.</p>
<p>Behavior changes that require this treatment include changing the default
value of an existing
configuration key, changing what an existing key’s values mean, and changing
the semantics of an
<code class="docutils literal notranslate"><span
class="pre">Incompatible</span></code> expression or operator whose divergence
from Spark users may have come to depend on.</p>
+<p>Changing a <code class="docutils literal notranslate"><span
class="pre">testing</span></code> or internal key is not a behavior change for
this purpose, and needs no escape
+hatch. See
+<a class="reference internal"
href="#testing-and-internal-configurations-are-exempt">Testing and Internal
Configurations Are Exempt</a>.</p>
<section id="lifetime-of-a-legacy-configuration">
<h3>Lifetime of a Legacy Configuration<a class="headerlink"
href="#lifetime-of-a-legacy-configuration" title="Link to this
heading">#</a></h3>
<p>A <code class="docutils literal notranslate"><span
class="pre">spark.comet.legacy.*</span></code> key is deprecated from the
moment it is added. Its purpose is to buy users
@@ -599,6 +657,9 @@ old key is kept working as a deprecated alias using the
<code class="docutils li
The alias may only be dropped in a major release.</p>
<p>Removing a configuration key outright requires a deprecation cycle: the key
must remain available,
with a deprecation warning, for at least one minor release before it is
removed in a major release.</p>
+<p>Neither rule applies to a <code class="docutils literal notranslate"><span
class="pre">testing</span></code> or internal key, which may be renamed without
an alias and
+removed in any release. See
+<a class="reference internal"
href="#testing-and-internal-configurations-are-exempt">Testing and Internal
Configurations Are Exempt</a>.</p>
</section>
</section>
<section id="correctness-fixes-are-not-breaking-changes">
@@ -834,7 +895,10 @@ incompatibilities.</p>
</div>
<nav class="bd-toc-nav page-toc"
aria-labelledby="pst-page-navigation-heading-2">
<ul class="visible nav section-nav flex-column">
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#comet-s-compatibility-surface">Comet’s Compatibility Surface</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#comet-s-compatibility-surface">Comet’s Compatibility Surface</a><ul
class="nav section-nav flex-column">
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#testing-and-internal-configurations-are-exempt">Testing and Internal
Configurations Are Exempt</a></li>
+</ul>
+</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#what-each-version-component-means">What Each Version Component
Means</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#major-releases">Major Releases</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#minor-releases">Minor Releases</a></li>
diff --git a/contributor-guide/config_conventions.html
b/contributor-guide/config_conventions.html
index 66a1c237f0..085e6b0245 100644
--- a/contributor-guide/config_conventions.html
+++ b/contributor-guide/config_conventions.html
@@ -606,18 +606,73 @@ be a red flag; both spell it <code class="docutils
literal notranslate"><span cl
The two do not have to match segment-for-segment — brevity in the symbol is
fine as long
as the key remains descriptive.</p>
</section>
-<section id="categories">
-<h2>Categories<a class="headerlink" href="#categories" title="Link to this
heading">#</a></h2>
+<section id="categories-and-visibility">
+<h2>Categories and Visibility<a class="headerlink"
href="#categories-and-visibility" title="Link to this heading">#</a></h2>
<p>Every <code class="docutils literal notranslate"><span
class="pre">ConfigEntry</span></code> must call <code class="docutils literal
notranslate"><span class="pre">.category(...)</span></code>. The category is
used to route the key into
the right table in the user guide’s <code class="docutils literal
notranslate"><span class="pre">configs.md</span></code>. Available categories
are declared as
<code class="docutils literal notranslate"><span
class="pre">CATEGORY_*</span></code> constants at the top of <code
class="docutils literal notranslate"><span
class="pre">CometConf.scala</span></code>. If a new config does not fit an
existing category, discuss adding a new one before landing the config.</p>
+<p>A second, independent choice is whether to call <code class="docutils
literal notranslate"><span class="pre">.internal()</span></code>, which keeps
the key out of
+<code class="docutils literal notranslate"><span
class="pre">configs.md</span></code> altogether.</p>
+<p>Neither choice is cosmetic. Together they decide whether Comet’s
+<a class="reference internal" href="../about/versioning_policy.html"><span
class="std std-doc">versioning policy</span></a> covers the key. The policy’s
rule keys on these
+two marks and nothing else: a key is exempt only if it is in the <code
class="docutils literal notranslate"><span class="pre">testing</span></code>
category or is marked
+<code class="docutils literal notranslate"><span
class="pre">internal()</span></code>, and every other <code class="docutils
literal notranslate"><span class="pre">spark.comet.*</span></code> key is
covered.</p>
+<div class="pst-scrollable-table-container"><table class="table">
+<thead>
+<tr class="row-odd"><th class="head"><p>Category</p></th>
+<th class="head"><p><code class="docutils literal notranslate"><span
class="pre">internal()</span></code></p></th>
+<th class="head"><p>Appears in <code class="docutils literal
notranslate"><span class="pre">configs.md</span></code></p></th>
+<th class="head"><p>Covered by the versioning policy</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p>any other</p></td>
+<td><p>no</p></td>
+<td><p>yes, in its category’s table</p></td>
+<td><p><strong>yes</strong></p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">testing</span></code></p></td>
+<td><p>no</p></td>
+<td><p>yes, under Development & Testing</p></td>
+<td><p>no</p></td>
+</tr>
+<tr class="row-even"><td><p>any</p></td>
+<td><p>yes</p></td>
+<td><p>no</p></td>
+<td><p>no</p></td>
+</tr>
+</tbody>
+</table>
+</div>
+<p>Note that a key read by string rather than through a registered <code
class="docutils literal notranslate"><span
class="pre">ConfigEntry</span></code> — as the
+per-expression <code class="docutils literal notranslate"><span
class="pre">spark.comet.expression.<Name>.allowIncompatible</span></code>
opt-ins are — has no way to carry
+either mark, so it is covered. Reaching for a dynamic key is therefore not a
way to avoid the
+guarantee; if the key is a debugging aid, give it a real <code class="docutils
literal notranslate"><span class="pre">ConfigEntry</span></code> in the <code
class="docutils literal notranslate"><span class="pre">testing</span></code>
category.</p>
+<p>Being covered commits the project to the key’s name, type, accepted values,
default, and semantics
+across minor releases. Being
+<a class="reference internal"
href="../about/versioning_policy.html#testing-and-internal-configurations-are-exempt"><span
class="std std-ref">exempt</span></a> means the key
+may be renamed, retyped, redefaulted, or removed in any release, including a
patch release, with no
+alias, no deprecation cycle, and no upgrade guide entry.</p>
+<p>So reach for <code class="docutils literal notranslate"><span
class="pre">testing</span></code> or <code class="docutils literal
notranslate"><span class="pre">internal()</span></code> only when the key
exists to let Comet’s own suites, or
+someone debugging Comet itself, reach a state the rest of the code is not
built to support —
+disabling native scans, running in on-heap mode, making a declined operator
throw. Use <code class="docutils literal notranslate"><span
class="pre">internal()</span></code>
+in particular when Comet should not publish the key at all, so that nobody can
adopt it in the first
+place.</p>
+<p>If a deployment would have a legitimate reason to set the key, it belongs
in a non-<code class="docutils literal notranslate"><span
class="pre">testing</span></code>
+category and must not be <code class="docutils literal notranslate"><span
class="pre">internal()</span></code>, and the guarantees come with it. Do not
use either mechanism
+as a way to ship a production knob without committing to it.</p>
</section>
<section id="renaming-an-existing-config">
<h2>Renaming an Existing Config<a class="headerlink"
href="#renaming-an-existing-config" title="Link to this heading">#</a></h2>
<p>Configs under <code class="docutils literal notranslate"><span
class="pre">spark.comet.*</span></code> are stable across minor releases: users
may have set them in
production <code class="docutils literal notranslate"><span
class="pre">spark-defaults.conf</span></code> files, Spark job submissions, or
notebooks. Renaming a key
must not silently break those deployments.</p>
+<p>Keys in the <code class="docutils literal notranslate"><span
class="pre">testing</span></code> category, and keys marked <code
class="docutils literal notranslate"><span
class="pre">internal()</span></code>, are
+<a class="reference internal"
href="../about/versioning_policy.html#testing-and-internal-configurations-are-exempt"><span
class="std std-ref">exempt from the versioning policy</span></a>
+and may be renamed outright: skip the <code class="docutils literal
notranslate"><span class="pre">withAlternative</span></code> call in step 1,
and step 5 with it. The
+rest of the checklist still applies, because a stale key string left behind in
code or docs is a
+bug either way.</p>
<p>Use the <code class="docutils literal notranslate"><span
class="pre">withAlternative</span></code> builder on <code class="docutils
literal notranslate"><span class="pre">ConfigBuilder</span></code> to keep the
old key working as a
deprecated alias:</p>
<div class="highlight-scala notranslate"><div
class="highlight"><pre><span></span><span class="kd">val</span><span class="w">
</span><span class="nc">COMET_FORCE_SHJ</span><span class="p">:</span><span
class="w"> </span><span class="nc">ConfigEntry</span><span
class="p">[</span><span class="nc">Boolean</span><span class="p">]</span><span
class="w"> </span><span class="o">=</span>
@@ -690,6 +745,10 @@ config only if the fix has an unusually wide blast radius,
and say so in the PR
<li><p><strong>Changes to which expressions and operators run
natively.</strong> Falling back to Spark, or ceasing
to, changes performance rather than results.</p></li>
</ul>
+<p>Nor does a change to a key in the <code class="docutils literal
notranslate"><span class="pre">testing</span></code> category or a key marked
<code class="docutils literal notranslate"><span
class="pre">internal()</span></code>, both
+<a class="reference internal"
href="../about/versioning_policy.html#testing-and-internal-configurations-are-exempt"><span
class="std std-ref">exempt from the versioning policy</span></a>:
+the default and the meaning may change in any release, with no legacy config
and no upgrade guide
+entry. Update the suites that set it in the same PR.</p>
<p>Removing a legacy config is a major-release change, handled the same way as
removing a deprecated
alias.</p>
</section>
@@ -744,7 +803,7 @@ alias.</p>
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#key-naming">Key Naming</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#symbol-naming-scala">Symbol Naming (Scala)</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#categories">Categories</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#categories-and-visibility">Categories and Visibility</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#renaming-an-existing-config">Renaming an Existing Config</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#changing-the-behavior-of-an-existing-config">Changing the Behavior of an
Existing Config</a></li>
</ul>
diff --git a/contributor-guide/development.html
b/contributor-guide/development.html
index dfddd8569a..e4200f0e7b 100644
--- a/contributor-guide/development.html
+++ b/contributor-guide/development.html
@@ -1085,6 +1085,11 @@ It is possible to debug both native and JVM code
concurrently as described in th
</section>
<section id="submitting-a-pull-request">
<h2>Submitting a Pull Request<a class="headerlink"
href="#submitting-a-pull-request" title="Link to this heading">#</a></h2>
+<p>Use <code class="docutils literal notranslate"><span class="pre">git</span>
<span class="pre">push</span></code> for normal updates to your PR branch. If
you need to force push after a rebase
+or amend, use <code class="docutils literal notranslate"><span
class="pre">git</span> <span class="pre">push</span> <span
class="pre">--force-with-lease</span></code> instead of <code class="docutils
literal notranslate"><span class="pre">git</span> <span class="pre">push</span>
<span class="pre">--force</span></code> (or <code class="docutils literal
notranslate"><span class="pre">-f</span></code>). This reduces
+the risk of accidentally overwriting another maintainer’s commits when
multiple people push
+to the same PR branch. If the lease check rejects the push, inspect and
integrate the remote
+changes before retrying; do not switch to <code class="docutils literal
notranslate"><span class="pre">--force</span></code> to bypass the check.</p>
<p>Before submitting a pull request, follow this checklist to ensure your
changes are ready:</p>
<section id="format-your-code">
<h3>1. Format Your Code<a class="headerlink" href="#format-your-code"
title="Link to this heading">#</a></h3>
diff --git a/searchindex.js b/searchindex.js
index 911d28dca5..419e14a3dd 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&":
[[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/":
[[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install
Comet": [[62, "install-comet"], [73, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&":
[[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/":
[[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install
Comet": [[62, "install-comet"], [73, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index c87f847ad2..1a8bde40c4 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -1153,6 +1153,17 @@ shuffle, including applications that rely only on remote
reliable storage or dec
</section>
<section id="development-testing-settings">
<h2>Development & Testing Settings<a class="headerlink"
href="#development-testing-settings" title="Link to this heading">#</a></h2>
+<p>These settings exist for Comet’s own test suites and for debugging. They
are <strong>not covered by the
+<a class="reference internal"
href="../../about/versioning_policy.html#testing-and-internal-configurations-are-exempt"><span
class="std std-ref">versioning policy</span></a></strong>:
+their names, defaults, accepted values, and meanings may change in any
release, including a patch
+release, and any of them may be removed without a deprecation cycle. Do not
set them in
+production.</p>
+<p>Comet also marks a handful of keys internal and deliberately leaves them
off this page entirely.
+They are maintainer escape hatches, not settings, and carry no guarantee of
any kind. Absence from
+this page does not by itself mean that, though: the per-expression
+<code class="docutils literal notranslate"><span
class="pre">spark.comet.expression.<Name>.allowIncompatible</span></code>
opt-ins are documented in the
+<a class="reference internal" href="compatibility/index.html"><span class="std
std-doc">compatibility guide</span></a> rather than here, and the versioning
policy covers
+them like any other production setting.</p>
<!--BEGIN:CONFIG_TABLE[testing]-->
<!-- prettier-ignore-start -->
<div class="pst-scrollable-table-container"><table class="table">
diff --git a/user-guide/latest/tuning.html b/user-guide/latest/tuning.html
index a85734b8e6..83f91ddf3e 100644
--- a/user-guide/latest/tuning.html
+++ b/user-guide/latest/tuning.html
@@ -1086,6 +1086,13 @@ direct-column <code class="docutils literal
notranslate"><span class="pre">IS</s
projects the file schema. The original null checks and residual runtime filter
remain in place.
The original join still verifies matches, including any hash collisions
admitted by the filter.
Standalone projections, other filter expressions, and limits prevent reader
attachment.</p>
+<p>To preserve schema-conversion and timestamp-overflow errors, runtime reader
pruning is disabled for
+each file whose projected or statically filtered columns require schema
adaptations beyond direct
+column mappings or literal values. This conservative check also disables
reader pruning for allowed
+<code class="docutils literal notranslate"><span
class="pre">INT32</span></code> to <code class="docutils literal
notranslate"><span class="pre">BIGINT</span></code> promotion and for
projecting a subset of a struct’s fields, even when those
+adaptations cannot fail. Nested column pruning still reads only the requested
struct fields. Scans
+with supplied file statistics also skip reader attachment. These cases still
use runtime filtering
+on decoded batches.</p>
<p>Filters stay within the task’s native plan and do not propagate across
Spark exchanges or JVM/Arrow
boundaries. A shuffled hash join can still filter probe batches after shuffle,
but it cannot send
its filter back to an earlier scan stage. Compare the <a class="reference
internal" href="metrics.html#hash-joins"><span class="std
std-ref">runtime-filter and scan metrics</span></a>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
