Jack created SPARK-52550:
----------------------------
Summary: SparkSessionExtensions requires support for DSV2 based
extensions which require the classic SparkSession
Key: SPARK-52550
URL: https://issues.apache.org/jira/browse/SPARK-52550
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 4.0.0
Reporter: Jack
Extensions providing connector capabilities such as
[https://github.com/apache/cassandra-spark-connector] register custom V2
strategies as part of the extension. However, in order to implement the data
source v2 strategy, we require a session type to be provided as
classic.SparkSession [due to API changes in
DSV2|https://github.com/apache/spark/commit/5db31aec33c53aaa7c814f33ec84e6ba66fc193b#diff-7aeb491d44e183c8c8cf86d90b57701dba009fc19983c2a5c09449c768b047ceR36].
It appears it is no longer possible to implement a custom strategy from an
extension in this mechanism, since SparkSessionExtensions only provides a
handle to the SparkSession class, which cannot be used for DSV2 strategy
planners.
This item of work is to enable a user to provision extensions when they know
the session will be scoped to classic, allowing registration of a DSV2
strategy, eg.
{code:java}
import org.apache.spark.sql.classic.{SparkSession => ClassicSparkSession}
.
.
.
case class MyCustomStrategy(spark: ClassicSparkSession) extends Strategy with
Serializable {
.
.
.
override def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
.
.
val dataSourceOptimizedPlan = new DataSourceV2Strategy(spark)...{code}
Where this is registered via
{code:java}
class CoolSparkExtensions extends (SparkSessionExtensions => Unit) with Logging
{
override def apply(extensions: SparkSessionExtensions): Unit = {
extensions.injectPlannerStrategy(MyCustomStrategy.apply)
.
.
.{code}
Worth noting that the existing API is marked both Experimental and @Unstable,
meaning changes proposed could be considered if a better solution to this issue
is not devised.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
