dawidwys commented on a change in pull request #7124: [FLINK-9574] [doc] Rework documentation for custom state serializers and state evolution URL: https://github.com/apache/flink/pull/7124#discussion_r235296908
########## File path: docs/dev/stream/state/schema_evolution.md ########## @@ -0,0 +1,92 @@ +--- +title: "State Schema Evolution" +nav-parent_id: streaming_state +nav-pos: 6 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +* ToC +{:toc} + +## Overview + +Apache Flink streaming applications are typically designed to run indefinitely for long periods of time. +As with all long-running services, the applications need to be updated to adapt to changing requirements. +This goes the same for data schemas that the applications work against; they evolve along with the application. + +This page provides an overview of how you can evolve your state type's data schema. +The current restrictions varies across different type's and state structures (et.c `ValueState`, `ListState`, etc.). + +Note that the information on this page is relevant only if you are using state serializers that is +generated by Flink's own [type serialization framework]({{ site.baseurl }}/dev/types_serialization.html). +That is, when declaring your state, the provided state descriptor is not configured to use a specific `TypeSerializer` +or `TypeInformation`, and therefore allowing Flink to infer information about the state type: + +<div data-lang="java" markdown="1"> +{% highlight java %} +ListStateDescriptor<MyPojoType> descriptor = + new ListStateDescriptor<>( + "state-name", + MyPojoType.class); + +checkpointedState = getRuntimeContext().getListState(descriptor); +{% endhighlight %} +</div> + +Under the hood, whether or not the schema of state can be evolved depends on the serializer used to read / write +persisted state bytes. Simply put, a registered state's schema can only be evolved if its serializer properly +supports it. This is handled transparently by serializers generated by Flink's type serialization framework +(current scope of support is listed [below]({{ site.baseurl }}/dev/stream/state/schema_evolution#supported-data-types-for-schema-evolution)). + +If you intend to implement a custom `TypeSerializer` for your state type and would like to learn how to implement +the serializer to support state schema evolution, please refer to +[Custom State Serialization]({{ site.baseurl }}/dev/stream/state/custom_serialization). +The documentation there also covers necessary internal details about the interplay between state serializers and Flink's +state backends to support state schema evolution. + +## Evolving state schema Review comment: How about adding a very high level description of how does the migration happens? Something along the lines that old serializer is used to read state, new serializer is used to write state with new schema. I would also mention here already that the compatibility check between those serializers is performed. Then I would link to more detailed description for particular state backends. I guess that you left it out, because you intended this page to be for "basic/simple" users, but I think if somebody ended up looking in for state migration he/she already has some deeper knowledge. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
