GitHub user xintongsong added a comment to the discussion: Short-Term Memory Design
3. Why not using state descriptor to distinguish memory entries. - Having multiple state descriptors is less efficient compared to multiple entries in a map state, in terms of per-state (descriptor) overhead, state / checkpoint management complexity, compression ratio, etc. - The flink community is planning to drop the supports for dynamically creating new states in runtime, because it leads to many problems in state management and compatibility control. Instead, we will require users to declare all the states to be used before executing the job. That might be not flexible enough for short-term memory, compared to a map state where you can add new k-v entries anytime you want. Why not using the list state - A map state can cover most use cases that other types of state (value, list, ...) can cover. It is more flexible to support both list-types and other types or named status & properties. - The map state comes with an advantage that when accessing an entry of the map we don't need to deserialize the whole state. - AFAIK, the only advantage of the list states is that, it has better supports for redistribution (split and merge), which is helpful when using as an operator state and when we change the parallelism of a Flink job. Which is not relevant to us because we are using keyed states. GitHub link: https://github.com/apache/flink-agents/discussions/40#discussioncomment-13728668 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
