kz930 commented on code in PR #8506:
URL: https://github.com/apache/texera/pull/8506#discussion_r4265220147
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/intersect/IntersectOpDesc.scala:
##########
@@ -52,4 +51,10 @@ class IntersectOpDesc extends LogicalOp {
inputPorts = List(InputPort(PortIdentity()), InputPort(PortIdentity(1))),
outputPorts = List(OutputPort(blocking = true))
)
+
+ // Distinct rows in both inputs. concat + duplicated (not pd.merge) so NaN ==
+ // NaN matches the JVM HashSet's null equality.
+ override def generateStandaloneCode(): String =
+ """_both = pd.concat([in1df.drop_duplicates(), in2df.drop_duplicates()],
ignore_index=True)
Review Comment:
@carloea2 Distinct had the same NaN and signed-zero gap, since its export
also used `drop_duplicates`. It is fixed in #8975 with the same row keys, and
opened separately because Distinct is already on main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
