carloea2 commented on code in PR #8506:
URL: https://github.com/apache/texera/pull/8506#discussion_r4254015345
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/hashJoin/HashJoinProbeOpExec.scala:
##########
@@ -80,10 +90,10 @@ class HashJoinProbeOpExec[K](
if (port == 0) {
// Load build hash map
val key = tuple.getField[K](HASH_JOIN_INTERNAL_KEY_NAME)
+ val names = tuple.getSchema.getAttributeNames.filterNot(n => n ==
HASH_JOIN_INTERNAL_KEY_NAME)
+ if (leftAttributeNames.isEmpty) leftAttributeNames = names
Review Comment:
The column names remain empty when this worker receives no build rows. In a
FULL_OUTER join with both schemas {k, x}, an unmatched right row then fills
left x and leaves x#@1 null. I reproduced this with an empty build side; the
export places the value correctly. A differently named right key also leaks
into a same-named left payload. Please pass the declared build schema
independently of its rows and cover empty build partitions.
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/intersect/IntersectOpDesc.scala:
##########
@@ -52,4 +51,10 @@ class IntersectOpDesc extends LogicalOp {
inputPorts = List(InputPort(PortIdentity()), InputPort(PortIdentity(1))),
outputPorts = List(OutputPort(blocking = true))
)
+
+ // Distinct rows in both inputs. concat + duplicated (not pd.merge) so NaN ==
+ // NaN matches the JVM HashSet's null equality.
+ override def generateStandaloneCode(): String =
+ """_both = pd.concat([in1df.drop_duplicates(), in2df.drop_duplicates()],
ignore_index=True)
Review Comment:
The new set exports still disagree with native equality. With one valid NaN
row on each side, native Intersect returns zero rows but this returns one.
Difference returns one natively and zero exported; Symmetric Difference returns
two natively and zero exported. Opposite signed zeros show the same
differences. Finite and null controls agree. Please align the semantics and
test both execution paths; nullable dtypes alone do not resolve this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
