Github user fhueske commented on the issue:
https://github.com/apache/flink/pull/3590
Hi @rtudoran,
thanks for doing the benchmark and posting the numbers! The recommended
state backend for production settings is the RocksDB backend (see
[production-readiness
docs](https://ci.apache.org/projects/flink/flink-docs-release-1.2/ops/production_ready.html#choice-of-state-backend)).
The in-memory backends store state as objects on the heap and can easily kill
the JVM with an OutOfMemoryError. Also the in-memory backends do not
de/serialize data, so there is not an actual advantage is using the MapState
which was mainly motivated by the reduced serialization effort. There are plans
to implement a state backend using managed memory (similar to the batch
algorithms). This backend would also serialize and deserialize data to/from
pre-allocated byte arrays. So optimizing for de/serialization makes sense, IMO.
The `KeyedOneInputStreamOperatorTestHarness` is part of the
`flink-streaming-java` test-jar artifact. This is added by adding the following
dependency to the `flink-table` `pom.xml`.
```<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.10</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
```
I'll have a detailed look at your PR tomorrow.
Best, Fabian
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
