We're experiencing the exact same issue while running load tests on Spark
2.3.1 with Structured Streaming and `mapGroupsWithState`.
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
-
To unsubscribe e-mail:
Hello Dev!
Spark structured streaming job with simple window aggregation is leaking
file descriptor on kubernetes as cluster manager setup. It seems bug.
I am suing HDFS as FS for checkpointing.
Have anyone observed same? Thanks for any help.
Please find more details in trailing email.
For
Hello All!
I am using spark 2.3.1 on kubernetes to run a structured streaming spark
job which read stream from Kafka , perform some window aggregation and
output sink to Kafka.
After job running few hours(5-6 hours), the executor pods is getting
crashed which is caused by "Too many open files in
