MartijnVisser commented on code in PR #29934:
URL: https://github.com/apache/flink/pull/29934#discussion_r4601309938
##########
docs/content/docs/ops/upgrading.md:
##########
@@ -176,16 +176,210 @@ When upgrading an application by changing its topology,
a few things need to be
### Table API & SQL
-Due to the declarative nature of Table API & SQL programs, the underlying
operator topology and state
-representation are mostly determined and optimized by the table planner.
+For Table API & SQL programs, the table planner decides the operator topology
and how each operator
+stores its state. What you can change while keeping the state depends on what
changes: the Flink
+version, the query, or a [process table function (PTF)]({{< ref
"docs/dev/table/functions/ptfs" >}})
+that the query calls.
+
+#### Upgrading the Flink Version
+
+A savepoint contains the state of each operator, but not the operators
themselves. A DataStream
+program builds the same operators again from its code when it restarts. A SQL
script or a Table API
+program contains only the query: the planner, which is part of Flink and not
of your JAR, turns it
+into operators again on every start. A newer Flink version can plan the same
query differently, for
+example because of new optimizer rules or a new version of an operator, and
the state in the
+savepoint then doesn't match the new operators.
+
+A [compiled plan]({{< ref "docs/concepts/sql-table-concepts/compiled_plan" >}})
+stores the result of the planning: the execution nodes of the pipeline with
their IDs and versions.
+A newer Flink version runs the plan as it is, as long as it still supports the
versions of the
+execution nodes in the plan, so that the operators find their state in the
savepoint.
+`COMPILE AND EXECUTE PLAN` compiles the plan only if the file doesn't exist
yet, and otherwise runs
+the plan from the file. Run the same statement on every Flink version:
+
+{{< tabs "upgrade-with-compiled-plan" >}}
+{{< tab "SQL CLI" >}}
+```sql
+-- 1. When you first deploy the job: create the tables and register the
functions
+-- that the query uses, then run
+COMPILE AND EXECUTE PLAN 's3://my-bucket/plans/my-job.json' FOR
+INSERT INTO sink
+SELECT * FROM MyPtf(input => TABLE src PARTITION BY user_id, uid => 'my-ptf');
+
+-- 2. Before the upgrade, stop the job with a savepoint. SHOW JOBS lists the
job ID,
+-- and STOP JOB prints the path of the savepoint.
+SET 'execution.checkpointing.savepoint-dir' = 's3://my-bucket/savepoints';
+STOP JOB '<job_id>' WITH SAVEPOINT;
+
+-- 3. On the new Flink version: create the same tables and register the same
functions,
+-- set the savepoint from step 2, and run the same statement.
+SET 'execution.state-recovery.path' =
's3://my-bucket/savepoints/savepoint-...';
+COMPILE AND EXECUTE PLAN 's3://my-bucket/plans/my-job.json' FOR
+INSERT INTO sink
+SELECT * FROM MyPtf(input => TABLE src PARTITION BY user_id, uid => 'my-ptf');
+-- The setting applies to every later job of the session.
+RESET 'execution.state-recovery.path';
+```
+{{< /tab >}}
+{{< tab "Java" >}}
+```java
+TableEnvironment tEnv =
TableEnvironment.create(EnvironmentSettings.inStreamingMode());
+tEnv.executeSql("CREATE TABLE src (...) WITH (...)");
+tEnv.executeSql("CREATE TABLE sink (...) WITH (...)");
+tEnv.createTemporarySystemFunction("MyPtf", MyPtf.class);
+
+tEnv.executeSql(
+ "COMPILE AND EXECUTE PLAN 's3://my-bucket/plans/my-job.json' FOR "
+ + "INSERT INTO sink "
+ + "SELECT * FROM MyPtf(input => TABLE src PARTITION BY
user_id, uid => 'my-ptf')");
+```
-Be aware that any change to both the query and the Flink version could lead to
state incompatibility.
-Every new major-minor Flink version (e.g. `1.12` to `1.13`) might introduce
new optimizer rules or more
-specialized runtime operators that change the execution plan. However, the
community tries to keep patch
-versions state-compatible (e.g. `1.13.1` to `1.13.2`).
+Instead of a file, the plan can also be part of the JAR of the job. Compile it
once, for example with
+`tEnv.compilePlanSql("INSERT INTO ...").writeToFile(...)`, add the file to the
resources of the
+project, and run it with:
+
+```java
+String plan;
+try (InputStream in = MyJob.class.getResourceAsStream("/plans/my-job.json")) {
+ plan = new String(in.readAllBytes(), StandardCharsets.UTF_8);
+}
+tEnv.loadPlan(PlanReference.fromJsonString(plan)).execute();
+```
+
+Before the upgrade, stop the job with a savepoint, and start the same JAR on
the new Flink version
+from it:
+
+```bash
+$ ./bin/flink stop --savepointPath s3://my-bucket/savepoints <job_id>
+$ ./bin/flink run --fromSavepoint s3://my-bucket/savepoints/savepoint-...
my-job.jar
+```
+{{< /tab >}}
+{{< /tabs >}}
+
+Keep the plan file as long as the job runs. Don't delete it, and don't compile
the plan again for an
+upgrade. A new compilation can plan the query differently, and the operator
IDs it assigns can
+differ from the ones in the savepoint.
+
+Before you run the plan on the new version, create its temporary tables and
functions again. Flink
+guarantees an upgrade by one minor version. See [Compiled Plans]({{< ref
"docs/concepts/sql-table-concepts/compiled_plan" >}}) for older plans, what a
+plan stores, and what can't be compiled.
+
+Without a compiled plan, a restore after a version upgrade can work, but there
is no guarantee.
+
+##### Jobs That Run Without a Compiled Plan
+
+The operator IDs of a job that runs from a compiled plan differ from the IDs
of the same job without
+one, except for the state of PTFs. A savepoint of a job without a compiled
plan therefore doesn't
+restore into the same job from a compiled plan: the restore fails on the
sources and the other
+stateful operators. That's why you compile the plan when you first deploy the
job. For a job that
+already runs without one, there are two options:
+
+1. **Upgrade without a compiled plan.** Stop the job with a savepoint, and
start the same SQL
+ statement or JAR on the new Flink version from it, with
`execution.state-recovery.path` or
+ `--fromSavepoint` as above. This works only if the new version plans the
query the same way.
+ Test it first: start the job on the new version from a copy of the
savepoint in a test
+ environment, with sinks that don't write to production, and check its
results. If the new
+ version plans the query differently, the restore fails, for example with
+ `Cannot map checkpoint/savepoint state for operator ...`. Keep the job on
the current version
+ until the test passes; the savepoint stays valid.
+2. **Move the job to a compiled plan.** This keeps the state of the PTFs only.
All other stateful
+ operators, such as joins and aggregations, start with empty state, and each
source starts from
+ its configured start position. Use it only if that state can be dropped,
for example for a job
+ that reads a source, calls a PTF, and writes to a sink. Do it on the
current Flink version, as a
+ separate step before the upgrade: set the start position of each source
(see the warning in the
+ next section), start the job with `COMPILE AND EXECUTE PLAN` from the
savepoint, and allow Flink
+ to skip the state that it can't map with `--allowNonRestoredState` (in SQL:
+ `SET 'execution.state-recovery.ignore-unclaimed-state' = 'true'`). From
then on, upgrade with the
+ compiled plan.
+
+#### Changing the Query
+
+When a job starts from a savepoint, Flink matches the state in the savepoint
to the operators of the
+job by operator ID, as for the [DataStream API](#matching-operator-state). In
a Table API or SQL
+program, only PTFs have an ID that you control:
+
+* **PTFs:** Flink restores the state of a PTF by its `uid` argument. Set the
`uid` explicitly, and
+ don't change it. Without it, the name of the function is used, and renaming
the function then
+ loses the state. See
+ [Query Evolution with UIDs]({{< ref "docs/dev/table/functions/ptfs"
>}}#query-evolution-with-uids).
+* **All other operators**, such as sources, sinks, joins, and aggregations,
get their IDs from the
+ planner. There is no equivalent of `uid()` or `setUidHash()` for them. A job
that runs from a
+ compiled plan uses the IDs in the plan. Without a plan, the IDs are derived
from the topology of
+ the job and change when the topology changes. The state of a source is its
read position.
+
+These changes keep the state of a PTF:
+
+* Filters and projections before or after the PTF.
+* A different sink with the same schema.
+* Different options of the source table, as long as its schema stays the same.
+* A different name of the function, as long as the `uid` stays the same.
+
+These changes fail the restore or lose the state of the PTF:
+
+* **A different or removed `uid`:** The restore fails with
+ `Cannot map checkpoint/savepoint state for operator ...`.
+* **A different `PARTITION BY` clause:** The PTF stores its state per key. If
the types of the key
+ change, the restore fails with `The new key serializer ... must be
compatible`. If the types stay
+ the same, the job restores without an error, but the state belongs to the
old keys and each new
+ key starts with empty state.
+
+The state of the other operators is kept only if their IDs stay the same. For
example, without a
+compiled plan, adding an `INSERT` statement to a statement set changes the ID
of a source that both
+statements read.
+
+A compiled plan pins the query. `COMPILE AND EXECUTE PLAN` doesn't use a
changed statement as long
+as the plan file exists. To change the query of a job that runs from a
compiled plan, compile the
+changed statement to a new file and run that file from the savepoint.
+
+The operators other than PTFs keep their state only if the new plan assigns
them the same IDs,
+which isn't guaranteed. The node IDs of a plan depend on the statements that
the same process
+planned before. Compile every plan of the job, including the first one, in a
new SQL Client or
+Table API process, not in a SQL Gateway or SQL Client session that has planned
other statements. Test the restore from a copy of the savepoint, and keep the
old plan file to go back to.
+
+{{< hint warning >}}
+When you use `--allowNonRestoredState` (in SQL:
`execution.state-recovery.ignore-unclaimed-state`)
+and it skips the state of a source, the source starts from its configured
start position. It can
+then read records again that the PTF has already processed, and the restored
state of the PTF counts
+these records twice. Before you skip the state of a source, set its start
position so that it
+continues where the old job stopped, for example with a timestamp or the
offsets of a consumer
+group.
+{{< /hint >}}
-See the [table state management section]({{< ref
"docs/concepts/sql-table-concepts/overview" >}}#state-management)
-for more information.
+Setting `table.exec.uid.generation` to `ALWAYS` doesn't give stable IDs
either: they depend on the
+statements that were planned before in the same process. Use a compiled plan
instead.
+
+#### Changing a Process Table Function
+
+You can change the logic of a PTF without losing its state. Flink restores
each state entry of a PTF
+by its name, and the state entry must keep its data type. Unlike POJO and Avro
types in the
+DataStream API, the data type of a PTF state entry doesn't evolve. Set the
name of every state entry
+that you want to keep explicitly with `@StateHint(name = "...")`. Otherwise,
the name is the name of
+the parameter of the `eval` method.
+
+| Change to the PTF
| Result |
+|:------------------------------------------------------------------------------|:-------|
+| Change the logic only
| Compatible. |
+| Add a state entry
| Compatible. The new state entry starts empty. |
+| Remove a state entry
| Compatible. Its state is no longer read. |
+| Change the order of the state parameters
| Compatible. |
+| Rename a state entry
| Not compatible. The state entry starts empty, without an error. |
+| Add a TTL with `@StateHint(ttl = "...")`
| Compatible. |
+| Add, remove, or change the type of a field of a POJO or `Row` state entry
| Not compatible. The restore fails with a `StateMigrationException`. A
`transient` field counts as removed. |
+| Rename a field of a POJO, or change the order of its fields or of the
parameters of its constructor | Not safe. The restore can succeed and assign
values to the wrong fields, without an error. |
+| Change a `ListView` or `MapView` to a POJO field, or a POJO to a `ValueView`
| Not compatible. The restore fails. |
+| Add an optional argument or an output column
| Compatible without a compiled plan. A compiled plan stores the signature of
the call, so the job needs a new plan, see [Changing the
Query](#changing-the-query). |
+
+To change the data type of a state entry, add a new state entry with the new
type and move the data
+from the old state entry to the new one in the `eval` method.
+
+The [PTF test harness]({{< ref "docs/dev/table/functions/ptfs"
>}}#testing-process-table-functions)
+tests the logic of a PTF, but it doesn't take or restore snapshots. To test
that a change keeps the
+state, take a savepoint of the current job in a test environment and start the
changed job from it.
+
+#### Parallelism
+
+You can change the parallelism of a Table API or SQL job between a savepoint
and a restore, with or
+without a compiled plan. As for DataStream programs, keep the maximum
parallelism the same.
Review Comment:
Done, and I removed the other DataStream references in this section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]