[
https://issues.apache.org/jira/browse/SPARK-15777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15590055#comment-15590055
]
Yan commented on SPARK-15777:
-----------------------------
There is a paragraph in the design doc about the ordering of rule application
as copied below:
-------------------------------------------------------------------------------------------------------------
5) Resolution/execution ordering of analyzer/optimizer/planner rules is as
follows:
a) For analyzer and planner, external rules precede the built-in rules;
b) For optimizer, the external rules follow the built-in rules.
This is following the existing ordering, but is subject to future revisits;
-------------------------------------------------------------------------------------------------------------
To prevent a custom rule from being too aggressive and modifying the plan
branches that are generic or specific to other data sources, guards can be
added against such intrusive rules. In the future even explicit ordering
specification might be added for more advanced plugged-in rules.
> Catalog federation
> ------------------
>
> Key: SPARK-15777
> URL: https://issues.apache.org/jira/browse/SPARK-15777
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Reporter: Reynold Xin
> Attachments: SparkFederationDesign.pdf
>
>
> This is a ticket to track progress to support federating multiple external
> catalogs. This would require establishing an API (similar to the current
> ExternalCatalog API) for getting information about external catalogs, and
> ability to convert a table into a data source table.
> As part of this, we would also need to be able to support more than a
> two-level table identifier (database.table). At the very least we would need
> a three level identifier for tables (catalog.database.table). A possibly
> direction is to support arbitrary level hierarchical namespaces similar to
> file systems.
> Once we have this implemented, we can convert the current Hive catalog
> implementation into an external catalog that is "mounted" into an internal
> catalog.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
