grundprinzip commented on code in PR #42099:
URL: https://github.com/apache/spark/pull/42099#discussion_r1270303330
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java:
##########
@@ -125,4 +125,26 @@ default CustomMetric[] supportedCustomMetrics() {
default CustomTaskMetric[] reportDriverMetrics() {
return new CustomTaskMetric[]{};
}
+
+ /**
+ * This enum defines how the columnar support for the partitions of the data
source
+ * should be determined. The default value is `PARTITION_DEFINED` which
indicates that each
+ * partition can deterine if it should be columnar or not. SUPPORTED and
UNSUPPORTED provide
+ * default shortcuts to indicate support for columnar data or not.
+ *
+ * @since 3.5.0
+ */
+ enum ColumnarSupportType {
+ PARTITION_DEFINED,
+ SUPPORTED,
+ UNSUPPORTED
+ }
+
+ /**
+ * Subclasses can implement this method to indicate if the support for
columnar data should
+ * be determined by each partition or is set as a default for the whole scan.
+ *
+ * @since 3.5.0
+ */
+ default ColumnarSupportType supportsColumnar() { return
ColumnarSupportType.PARTITION_DEFINED; }
Review Comment:
it can't be a boolean because there are three different values.
Since this is used as an override, we must preserve today's behavior for
backwards compatibility. The backward compatible behavior is to iterate over
all partitions to check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
