I'm trying to list partitions for a table with a filter using the below
function,
spark.sharedState.externalCatalog.listPartitionsByFilter()
Signature of listPartitionsByFilter in
org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener is as
below,
def listPartitionsByFilter(
db: String,
table: String,
predicates: Seq[Expression],
defaultTimeZoneId: String): Seq[CatalogTablePartition] = {
This is how I'm creating an expression to pass for the predicates parameter,
val exp = expr("`year`='2020' and `month`='03' and `day`='01'").expr
InterpretedPredicate.create(exp.expr).expression
spark.sharedState.externalCatalog.listPartitionsByFilter("prejoin",
"bookings", Seq(exp),"UTC")
This produces below error:
java.lang.UnsupportedOperationException: Cannot evaluate expression: 'year
at
org.apache.spark.sql.catalyst.expressions.Unevaluable$class.eval(Expression.scala:255)
However with empty sequence for predicates parameter, it produces results
spark.sharedState.externalCatalog.listPartitionsByFilter("prejoin",
"bookings", Seq.empty,"UTC")
Can someone help with correct way of creating predicates for
listPartitionsByFilter function?
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
