carloea2 commented on code in PR #8340:
URL: https://github.com/apache/texera/pull/8340#discussion_r3973442897
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/hashJoin/HashJoinOpDesc.scala:
##########
@@ -200,4 +200,28 @@ class HashJoinOpDesc[K] extends LogicalOp {
),
outputPorts = List(OutputPort())
)
+
+ // Equi-join: drop the probe key (kept only when its name differs from the
+ // build key), suffix colliding right columns "#@1" — matches JoinUtils.
Known
+ // Texera divergences: row order, null keys (NaN != NaN in merge), outer
+ // anti-row column placement.
+ override def generateStandaloneCode(): String = {
+ val buildKeyLit = objectMapper.writeValueAsString(buildAttributeName)
+ val probeKeyLit = objectMapper.writeValueAsString(probeAttributeName)
+ val how = joinType match {
+ case JoinType.INNER => "inner"
+ case JoinType.LEFT_OUTER => "left"
+ case JoinType.RIGHT_OUTER => "right"
+ case JoinType.FULL_OUTER => "outer"
+ }
+ val merge =
+ s"""out1df = in1df.merge(in2df, how=${pyStringLiteral(how)},
left_on=$buildKeyLit, """ +
+ s"""right_on=$probeKeyLit, suffixes=("", "#@1"))"""
+ val tail =
+ if (buildAttributeName != probeAttributeName)
+ s"out1df = out1df.drop(columns=[$probeKeyLit]).reset_index(drop=True)"
Review Comment:
This can drop a left payload column instead of the right join key. With left
`{id: 1, key: "payload"}` and right `{key: 1, value: 9}`, joining left `id` to
right `key` removes the payload and retains `key#@1`. JoinUtils preserves the
left payload. Please account for the renamed right key before dropping it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
