sunchao commented on code in PR #36697:
URL: https://github.com/apache/spark/pull/36697#discussion_r885234555
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanPartitioning.scala:
##########
@@ -32,15 +32,15 @@ import
org.apache.spark.util.collection.Utils.sequenceToOption
*/
object V2ScanPartitioning extends Rule[LogicalPlan] with SQLConfHelper {
override def apply(plan: LogicalPlan): LogicalPlan = plan transformDown {
- case d @ DataSourceV2ScanRelation(relation, scan:
SupportsReportPartitioning, _, _) =>
+ case d @ DataSourceV2ScanRelation(relation, scan:
SupportsReportPartitioning, _, None) =>
val funCatalogOpt = relation.catalog.flatMap {
case c: FunctionCatalog => Some(c)
case _ => None
}
val catalystPartitioning = scan.outputPartitioning() match {
case kgp: KeyGroupedPartitioning => sequenceToOption(kgp.keys().map(
- V2ExpressionUtils.toCatalyst(_, relation, funCatalogOpt)))
+ V2ExpressionUtils.toCatalystOpt(_, relation, funCatalogOpt)))
Review Comment:
I'm also a bit more inclined to use warning messages here. To debug
performance issue, it should be relatively straightforward to check out the
query plan and see whether there's any shuffle in it, combined with checking
query logs. In addition, existing V2 data source tables could already have
custom transforms such as bucket, days, etc, and we don't want queries against
these tables start to fail (because there is no function catalog) after
upgrading to Spark 3.3.
There is already warning message when function catalog exists but the
transform function doesn't, in the method `loadV2FunctionOpt`. It seems we
don't have warning message for the case when a custom transform function is
used but there is no function catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
