Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/3269#discussion_r37612164
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -93,3 +98,18 @@ abstract class PrunedScan extends BaseRelation {
abstract class PrunedFilteredScan extends BaseRelation {
def buildScan(requiredColumns: Array[String], filters: Array[Filter]):
RDD[Row]
}
+
+/**
+ * ::Experimental::
+ * An interface for experimenting with a more direct connection to the
query planner. Compared to
+ * [[PrunedFilteredScan]], this operator receives the raw expressions from
the
+ * [[org.apache.spark.sql.catalyst.plans.logical.LogicalPlan]]. Unlike
the other APIs this
+ * interface is not designed to be binary compatible across releases and
thus should only be used
+ * for experimentation.
+ */
+@Experimental
+abstract class CatalystScan extends BaseRelation {
+ def buildScan(requiredColumns: Seq[Attribute], filters:
Seq[Expression]): RDD[Row]
+}
--- End diff --
Do we really need this? Currently none of our build-in data source use this
and there is no example to demonstrate how to use it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
