I use the state function flatmapgroupswithstate to track state of a kafka stream. To further customize the state function I like to use a static datasource (JDBC) in the state function. This datasource contains data I like to join with the stream (as Iterator) within flatmapgroupswithstate.
When I try to access the JDBC source within flatmapgroupswithstate Spark execution freezes without any Exceptions or logging. To verify the JDBC connection works, I also tried to access the source outside the state function and that works. So now I join the static source with streaming source before feeding it to flatmapgroupswithstate. It seems to work so far… Any ideas why accessing the JDBC source within flatmapgroupswithstate could fail (freezes Spark execution)? Is it wise to use external datasources within flatmapgroupswithstate? Thanks, Chris
