kz930 commented on code in PR #8506:
URL: https://github.com/apache/texera/pull/8506#discussion_r4254736624
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/hashJoin/HashJoinProbeOpExec.scala:
##########
@@ -80,10 +90,10 @@ class HashJoinProbeOpExec[K](
if (port == 0) {
// Load build hash map
val key = tuple.getField[K](HASH_JOIN_INTERNAL_KEY_NAME)
+ val names = tuple.getSchema.getAttributeNames.filterNot(n => n ==
HASH_JOIN_INTERNAL_KEY_NAME)
+ if (leftAttributeNames.isEmpty) leftAttributeNames = names
Review Comment:
Fixed in ce2b89326: the worker now records each input port's declared schema
on the executor before that port finishes, and the probe reads the left names
from it instead of from the first build row. Both cases you named have an
empty-build-side test, and both fail without the fix.
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/intersect/IntersectOpDesc.scala:
##########
@@ -52,4 +51,10 @@ class IntersectOpDesc extends LogicalOp {
inputPorts = List(InputPort(PortIdentity()), InputPort(PortIdentity(1))),
outputPorts = List(OutputPort(blocking = true))
)
+
+ // Distinct rows in both inputs. concat + duplicated (not pd.merge) so NaN ==
+ // NaN matches the JVM HashSet's null equality.
+ override def generateStandaloneCode(): String =
+ """_both = pd.concat([in1df.drop_duplicates(), in2df.drop_duplicates()],
ignore_index=True)
Review Comment:
Fixed in 77c64a309: the three exports now select rows by a key that follows
the engine, so a NaN matches nothing and 0.0 and -0.0 stay apart. Each operator
has a parity test against its native executor, and all three fail on the old
exports.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
