[
https://issues.apache.org/jira/browse/KAFKA-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235814#comment-15235814
]
Michael Noll edited comment on KAFKA-3522 at 4/11/16 7:41 PM:
--------------------------------------------------------------
[~jkreps] wrote:
{quote}
I think you could potentially just write some version file on startup of the
job and in the future we could check that and if it doesn't match the current
version just delete the data on disk and let it rebuild.
{quote}
Could this delete-data-if-stored-version-is-older behavior cause issues when
e.g. performing rolling upgrades of Streams applications (I think it would be
safe, as we're only purging local state)? Also, what about the impacts of this
behavior on state replicas, i.e. shadow copies of state stores; could the
versions of these replicas become out-of-sync and thereby cause problems?
was (Author: miguno):
[~jkreps] wrote:
{quote}
I think you could potentially just write some version file on startup of the
job and in the future we could check that and if it doesn't match the current
version just delete the data on disk and let it rebuild.
{quote}
Could this delete-data-if-stored-version-is-older behavior cause issues when
e.g. performing rolling upgrades of Streams applications (I think it would be
safe, as we're only purging local state)? Also, what about the impacts of
state replicas, i.e. shadow copies of state stores; could the versions of
these replicas become out-of-sync and thereby cause problems?
> Consider adding version information into rocksDB storage format
> ---------------------------------------------------------------
>
> Key: KAFKA-3522
> URL: https://issues.apache.org/jira/browse/KAFKA-3522
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: Guozhang Wang
> Labels: semantics
> Fix For: 0.10.1.0
>
>
> Kafka Streams does not introduce any modifications to the data format in the
> underlying Kafka protocol, but it does use RocksDB for persistent state
> storage, and currently its data format is fixed and hard-coded. We want to
> consider the evolution path in the future we we change the data format, and
> hence having some version info stored along with the storage file / directory
> would be useful.
> And this information could be even out of the storage file; for example, we
> can just use a small "version indicator" file in the rocksdb directory for
> this purposes. Thoughts? [~enothereska] [~jkreps]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
