c21 commented on a change in pull request #29724:
URL: https://github.com/apache/spark/pull/29724#discussion_r487206481
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala
##########
@@ -56,8 +56,8 @@ import org.apache.spark.util.{CompletionIterator,
SerializableConfiguration}
* - Apply the optional condition to filter the joined rows as the final
output.
*
* If a timestamp column with event time watermark is present in the join keys
or in the input
- * data, then the it uses the watermark figure out which rows in the buffer
will not join with
- * and the new data, and therefore can be discarded. Depending on the provided
query conditions, we
+ * data, then it uses the watermark figure out which rows in the buffer will
not join with
Review comment:
@cloud-fan - updated.
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala
##########
@@ -246,13 +244,14 @@ case class StreamingSymmetricHashJoinExec(
// Join one side input using the other side's buffered/state rows. Here
is how it is done.
//
- // - `leftJoiner.joinWith(rightJoiner)` generates all rows from matching
new left input with
Review comment:
@cloud-fan , @HeartSaVioR - yes, this is just updating the comment,
because there's no `leftJoiner`/`rightJoiner`/`joinWith` in the file, and the
original author (https://github.com/apache/spark/pull/19271) should mean to
refer to `leftSideJoiner`/`rightSideJoiner`/`storeAndJoinWithOtherSide`. I
think it would make sense to be consistent between code and comment here. This
is anyway a minor change for comment only.
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala
##########
@@ -189,11 +189,9 @@ case class StreamingSymmetricHashJoinExec(
override def outputPartitioning: Partitioning = joinType match {
case _: InnerLike =>
PartitioningCollection(Seq(left.outputPartitioning,
right.outputPartitioning))
- case LeftOuter => PartitioningCollection(Seq(left.outputPartitioning))
- case RightOuter => PartitioningCollection(Seq(right.outputPartitioning))
- case x =>
- throw new IllegalArgumentException(
- s"${getClass.getSimpleName} should not take $x as the JoinType")
+ case LeftOuter => left.outputPartitioning
+ case RightOuter => right.outputPartitioning
+ case _ => throwBadJoinTypeException()
Review comment:
@xuanyuanking - sorry that I don't get how to change, are you suggesting
to have a string val for error message to be used in `throwBadJoinTypeException
` and `require(...)`: `val errorMessageForJoinType =
s"${getClass.getSimpleName} should not take $joinType as the JoinType")`, or
something else?
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala
##########
@@ -189,11 +189,9 @@ case class StreamingSymmetricHashJoinExec(
override def outputPartitioning: Partitioning = joinType match {
case _: InnerLike =>
PartitioningCollection(Seq(left.outputPartitioning,
right.outputPartitioning))
- case LeftOuter => PartitioningCollection(Seq(left.outputPartitioning))
- case RightOuter => PartitioningCollection(Seq(right.outputPartitioning))
- case x =>
- throw new IllegalArgumentException(
- s"${getClass.getSimpleName} should not take $x as the JoinType")
+ case LeftOuter => left.outputPartitioning
+ case RightOuter => right.outputPartitioning
+ case _ => throwBadJoinTypeException()
Review comment:
@xuanyuanking - sure, updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
