[
https://issues.apache.org/jira/browse/SPARK-26249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunitha Kambhampati updated SPARK-26249:
----------------------------------------
Description:
+Motivation:+
Spark has extension points API to allow third parties to extend Spark with
custom optimization rules. The current API does not allow fine grain control
on when the optimization rule will be exercised.
In our use cases, we have optimization rules that we want to add as extensions
to a batch in a specific order.
In the current API, there is no way to add a batch to the optimization using
the SparkSessionExtensions API, similar to the postHocOptimizationBatches in
SparkOptimizer.
+Proposal:+
Add 2 new API's to the existing Extension Points to allow for more flexibility
for third party users of Spark.
# Inject a optimizer rule to a batch in order
# Inject a optimizer batch in order
The design spec is
[here|[https://drive.google.com/file/d/1m7rQZ9OZFl0MH5KS12CiIg3upLJSYfsA/view?usp=sharing]]
was:
+Motivation:+
Spark has extension points API to allow third parties to extend Spark with
custom optimization rules. The current API does not allow fine grain control
on when the optimization rule will be exercised.
In our use cases, we have optimization rules that we want to add as extensions
to a batch in a specific order.
In the current API, there is no way to add a batch to the optimization using
the SparkSessionExtensions API, similar to the postHocOptimizationBatches in
SparkOptimizer.
+Proposal:+
Add 2 new API's to the existing Extension Points to allow for more flexibility
for third party users of Spark.
# Inject a optimizer rule to a batch in order
# Inject a optimizer batch in order
The design spec is
[here|[https://drive.google.com/file/d/1m7rQZ9OZFl0MH5KS12CiIg3upLJSYfsA/view?usp=sharing]]
> Extension Points Enhancements to inject a rule in order and to add a batch
> --------------------------------------------------------------------------
>
> Key: SPARK-26249
> URL: https://issues.apache.org/jira/browse/SPARK-26249
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Sunitha Kambhampati
> Priority: Major
>
> +Motivation:+
> Spark has extension points API to allow third parties to extend Spark with
> custom optimization rules. The current API does not allow fine grain control
> on when the optimization rule will be exercised.
> In our use cases, we have optimization rules that we want to add as
> extensions to a batch in a specific order.
> In the current API, there is no way to add a batch to the optimization using
> the SparkSessionExtensions API, similar to the postHocOptimizationBatches in
> SparkOptimizer.
> +Proposal:+
> Add 2 new API's to the existing Extension Points to allow for more
> flexibility for third party users of Spark.
> # Inject a optimizer rule to a batch in order
> # Inject a optimizer batch in order
> The design spec is
> [here|[https://drive.google.com/file/d/1m7rQZ9OZFl0MH5KS12CiIg3upLJSYfsA/view?usp=sharing]]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
