[ https://issues.apache.org/jira/browse/SPARK-38809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17518344#comment-17518344 ]
Anish Shrigondekar edited comment on SPARK-38809 at 4/6/22 6:38 PM: -------------------------------------------------------------------- Working on this PR and will send the change out soon. CC - [~kabhwan] was (Author: JIRAUSER287599): Working on this PR and will send the change out soon > Implement option to skip null values in symmetric hash impl of stream-stream > joins > ---------------------------------------------------------------------------------- > > Key: SPARK-38809 > URL: https://issues.apache.org/jira/browse/SPARK-38809 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 3.2.1 > Reporter: Anish Shrigondekar > Priority: Major > > Implement option to skip null values in symmetric hash impl of stream-stream > joins > * In the symmetric has join state manager, we can receive entries with null > values for a key and that caused the `removeByValue` and get iterators to > fail and run into the NullPointerException. > * This is possible if the state recovered is written from a old spark > version or its corrupted on disk. Since we don't have a utility to query this > state, we would like to provide a conf option to skip nulls for the symmetric > hash impl in stream stream joins. > -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org