[ https://issues.apache.org/jira/browse/SPARK-23869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
bharath kumar avusherla closed SPARK-23869. ------------------------------------------- > Spark 2.3.0 left outer join not emitting null values instead waiting for the > record in other stream > --------------------------------------------------------------------------------------------------- > > Key: SPARK-23869 > URL: https://issues.apache.org/jira/browse/SPARK-23869 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.3.0 > Reporter: bharath kumar avusherla > Priority: Major > > Left outer join on two streams not emitting the null outputs. It is just > waiting for the record to be added to other stream. Used socketstream to test > this. In our case we want to emit the records with null values which doesn't > match with id or/and not fall in time range condition > Details of the watermarks and intervals are: > val ds1Map = ds1 > .selectExpr("Id AS ds1_Id", "ds1_timestamp") > .withWatermark("ds1_timestamp","10 seconds") > val ds2Map = ds2 > .selectExpr("Id AS ds2_Id", "ds2_timestamp") > .withWatermark("ds2_timestamp", "20 seconds") > val output = ds1Map.join( ds2Map, > expr( > """ ds1_Id = ds2_Id AND ds2_timestamp >= ds1_timestamp AND ds2_timestamp <= > ds1_timestamp + interval 1 minutes """), > "leftOuter") > val query = output.select("*") > .writeStream > .outputMode(OutputMode.Append) > .format("console") > .option("checkpointLocation", "./ewe-spark-checkpoints/") > .start() > query.awaitTermination() > Thank you. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org