Hi Felipe Generally speaking, the key difference which impacts the performance is where they store data within windows. For Flink, it would store data and its related time-stamp within windows in state backend[1]. Once window is triggered, it would pull all the stored timer with coupled record-key, and then use the record-key to query state backend for next actions.
For Spark, first of all, we would talk about structured streaming [2] as it's better than previous spark streaming especially on window scenario. Unlike Flink built-in supported rocksDB state backend, Spark has only one implementation of state store providers. It's HDFSBackedStateStoreProvider which stores all of the data in memory, what is a very memory consuming approach and might come across OOM errors[3][4][5]. To avoid this, Databricks Runtime offer a 'RocksDBStateStoreProvider' but not open-source. We're lucky that open-source Flink already offers built-in RocksDB state backend to avoid OOM problem. Moreover, Flink community recently are developing spill-able memory state backend [7]. [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/state_backends.html [2] https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#window-operations-on-event-time [3] https://medium.com/@chandanbaranwal/state-management-in-spark-structured-streaming-aaa87b6c9d31 [4] http://apache-spark-user-list.1001560.n3.nabble.com/use-rocksdb-for-spark-structured-streaming-SSS-td34776.html#a34779 [5] https://github.com/chermenin/spark-states [6] https://docs.databricks.com/spark/latest/structured-streaming/production.html#optimize-performance-of-stateful-streaming-queries [7] https://issues.apache.org/jira/browse/FLINK-12692 Best Yun Tang ________________________________ From: Felipe Gutierrez <felipe.o.gutier...@gmail.com> Sent: Thursday, October 10, 2019 20:39 To: user <user@flink.apache.org> Subject: Difference between windows in Spark and Flink Hi all, I am trying to think about the essential differences between operators in Flink and Spark. Especially when I am using Keyed Windows then a reduce operation. In Flink we develop an application that can logically separate these two operators. Thus after a keyed window I can use .reduce/aggregate/fold/apply() functions [1]. In Spark we have window/reduceByKeyAndWindow functions which to me appears it is less flexible in the options to use with a keyed window operation [2]. Moreover, when these two applications are deployed in a Flink and Spark cluster respectively, what are the differences between their physical operators running in the cluster? [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#windows [2] https://spark.apache.org/docs/latest/streaming-programming-guide.html#window-operations Thanks, Felipe -- -- Felipe Gutierrez -- skype: felipe.o.gutierrez -- https://felipeogutierrez.blogspot.com
