[
https://issues.apache.org/jira/browse/FLINK-6761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-6761:
----------------------------------
Labels: auto-deprioritized-critical stale-major (was:
auto-deprioritized-critical)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Limitation for maximum state size per key in RocksDB backend
> ------------------------------------------------------------
>
> Key: FLINK-6761
> URL: https://issues.apache.org/jira/browse/FLINK-6761
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
> Affects Versions: 1.2.1, 1.3.0
> Reporter: Stefan Richter
> Priority: Major
> Labels: auto-deprioritized-critical, stale-major
>
> RocksDB`s JNI bridge allows for putting and getting {{byte[]}} as keys and
> values.
> States that internally use RocksDB's merge operator, e.g. {{ListState}}, can
> currently merge multiple {{byte[]}} under one key, which will be internally
> concatenated to one value in RocksDB.
> This becomes problematic, as soon as the accumulated state size under one key
> grows larger than {{Integer.MAX_VALUE}} bytes. Whenever Java code tries to
> access a state that grew beyond this limit through merging, we will encounter
> an {{ArrayIndexOutOfBoundsException}} at best and a segfault at worst.
> This behaviour is problematic, because RocksDB silently stores states that
> exceed this limitation, but on access (e.g. in checkpointing), the code fails
> unexpectedly.
> I think the only proper solution to this is for RocksDB's JNI bridge to build
> on {{(Direct)ByteBuffer}} - which can go around the size limitation - as
> input and output types, instead of simple {{byte[]}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
