szehon-ho commented on code in PR #58145:
URL: https://github.com/apache/spark/pull/58145#discussion_r3883048948
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableWithTableSample.scala:
##########
@@ -367,3 +371,69 @@ class InMemoryTableWithLegacyTableSample(
seed: Long): Boolean = true
}
}
+
+/**
+ * Sample table plus Catalyst advisory filters. V1 `SupportsPushDownFilters`
cannot
+ * mix with `SupportsPushDownCatalystFilters` (`pushedFilters` return types
clash),
+ * so this wraps the sample builder. The advisory SQL is specified by the
`advisory-filter`
+ * property.
+ */
+class InMemoryTableWithTableSampleAndAdvisoryFilters(
+ name: String,
+ columns: Array[Column],
+ partitioning: Array[Transform],
+ properties: util.Map[String, String])
+ extends InMemoryTableWithTableSample(name, columns, partitioning,
properties) {
+
+ override def newScanBuilder(options: CaseInsensitiveStringMap): ScanBuilder
= {
+ new SampleAndAdvisoryScanBuilder(new
InMemoryTableSampleScanBuilder(schema, options), schema)
+ }
+
+ private class SampleAndAdvisoryScanBuilder(
+ inner: InMemoryTableSampleScanBuilder,
+ tableSchema: StructType)
+ extends ScanBuilder
+ with SupportsPushDownTableSample
+ with SupportsPushDownCatalystFilters {
+
+ override def pushTableSample(
+ lowerBound: Double,
+ upperBound: Double,
+ withReplacement: Boolean,
+ seed: Long): Boolean =
+ inner.pushTableSample(lowerBound, upperBound, withReplacement, seed)
+
+ override def pushTableSample(
+ lowerBound: Double,
+ upperBound: Double,
+ withReplacement: Boolean,
+ seed: Long,
+ sampleMethod: SampleMethod): Boolean =
+ inner.pushTableSample(lowerBound, upperBound, withReplacement, seed,
sampleMethod)
+
+ override def pushFilters(filters: Seq[Expression]): Seq[Expression] = Nil
Review Comment:
Fixed in 0846449b10e. The wrapper now returns the Catalyst filters as
post-scan filters, so Spark evaluates the user predicate. The regression now
uses `WHERE id >= 3` and expects only rows 3 through 5, proving rows 1 and 2
are removed while TABLESAMPLE and advisory-filter handling still work. The
focused test passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
