kz930 commented on code in PR #8980:
URL: https://github.com/apache/texera/pull/8980#discussion_r4364588198
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/filter/SpecializedFilterOpDesc.scala:
##########
@@ -63,42 +64,183 @@ class SpecializedFilterOpDesc extends FilterOpDesc with
StandaloneCodeGenerator
)
}
- override def generateStandaloneCode(): String = {
+ override def generateStandaloneCode(): String =
generateStandaloneCode(Map.empty)
+
+ override def generateStandaloneCode(inputSchemas: Map[PortIdentity,
Schema]): String = {
// No predicate keeps no row: the executor's filter is `predicates.exists`,
// which answers false on an empty list. Passing the frame through would be
// the opposite answer.
if (predicates.isEmpty) return "out1df = in1df.iloc[0:0].copy()"
+ val declared: Map[String, AttributeType] = inputSchemas.values.headOption
+ .map(_.getAttributes.map(a => a.getName -> a.getType).toMap)
+ .getOrElse(Map.empty)
val conditions = predicates.map { p =>
val colLit = pyStringLiteral(p.attribute)
p.condition match {
case ComparisonType.IS_NULL => s"""(in1df[$colLit].isna())"""
case ComparisonType.IS_NOT_NULL => s"""(in1df[$colLit].notna())"""
- case other =>
- val op = other.getName // returns "=", ">=", "<", etc. (see
ComparisonType.java)
- val pyOp = if (op == "=") "==" else op
- // notna mirrors FilterPredicate, which answers false for every
condition
- // but IS_NULL / IS_NOT_NULL once the field is null. Only `!=` needs
it —
- // pandas answers True there, where every other operator answers
False —
- // but guarding all of them keeps the one rule visible in one place.
- s"""(in1df[$colLit].notna() & (in1df[$colLit] $pyOp
${coerceValue(p.value)}))"""
+ case other =>
+ // Without a schema the column's dtype stands in for its type.
+ val kind = declared.get(p.attribute).map(t =>
pyStringLiteral(t.name)).getOrElse("None")
Review Comment:
Thanks, the declared type now comes from the same case-insensitive Schema
lookup the engine uses, and your LONG Key/KEY case is a native-versus-export
test in this spec (fcff3cd64). The assertion from #9035 is updated for the new
helper where the two are combined, and the whole spec passes with #8597 and
#9035 merged in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
