dongjoon-hyun opened a new pull request, #547:
URL: https://github.com/apache/spark-connect-swift/pull/547
### What changes were proposed in this pull request?
This PR proposes to assign a unique plan ID to the root relation of every
`DataFrame`.
- A file-scope `Atomic<Int64>` generator hands out monotonically increasing
plan IDs, like the
existing `lambdaVariableID` in `HigherOrderFunctions.swift`.
- Both SQL-based initializers now delegate to `init(spark:plan:)`, so every
`DataFrame` goes
through a single place that stamps the plan ID.
- An existing plan ID is preserved. For example, `toDF()` without arguments
reuses `self.plan`,
and keeping its ID is what lets column references of the original
`DataFrame` keep working.
This mirrors Scala's `SparkSession.newDataset`, which only tags newly
created roots.
- `DataFrameInternalTests.removeCachedRemoteRelation` now disables the
server-side plan cache,
because a plan cached under the `DataFrame`'s plan ID otherwise keeps
answering `count()`
after the cached remote relation is removed.
### Why are the changes needed?
The server uses plan IDs to resolve column references that are bound to a
specific `DataFrame`,
which is how ambiguous columns of a self-join are disambiguated. This
mechanism landed in Apache
Spark [v3.4.0](https://github.com/apache/spark/releases/tag/v3.4.0)
(2023-04-07) via SPARK-41812
(server and PySpark) and SPARK-41823 (Scala client), both backported to
`branch-3.4`. This PR is
the prerequisite for supporting `DataFrame`-bound column accessors in this
client.
### Does this PR introduce _any_ user-facing change?
No new API.
However, since relations now carry plan IDs, the server-side session plan
cache (SPARK-47818,
available since Apache Spark 4.0.0) applies to this client for the first
time, exactly as it
already does for the Scala and Python Connect clients. Concretely, when a
single `DataFrame`
object created by `spark.read.<format>(path)` is reused after new files are
appended to that
path, it reports the result of its first execution instead of re-listing the
files. `DataFrame`s
of catalog tables, SQL queries, and temp views stay fresh, because only the
unresolved plan is
cached for them. This behavior was verified to be identical to the Scala
Connect client on both
Spark 4.0.4 and 4.3.0 servers.
### How was this patch tested?
Pass the CIs with a new `DataFrameInternalTests.planID` test.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
