twalthr commented on code in PR #29934:
URL: https://github.com/apache/flink/pull/29934#discussion_r4601434159
##########
docs/content/docs/ops/upgrading.md:
##########
@@ -176,16 +176,210 @@ When upgrading an application by changing its topology,
a few things need to be
### Table API & SQL
-Due to the declarative nature of Table API & SQL programs, the underlying
operator topology and state
-representation are mostly determined and optimized by the table planner.
+For Table API & SQL programs, the table planner decides the operator topology
and how each operator
+stores its state. What you can change while keeping the state depends on what
changes: the Flink
+version, the query, or a [process table function (PTF)]({{< ref
"docs/dev/table/functions/ptfs" >}})
+that the query calls.
+
+#### Upgrading the Flink Version
+
+A savepoint contains the state of each operator, but not the operators
themselves. A DataStream
+program builds the same operators again from its code when it restarts. A SQL
script or a Table API
+program contains only the query: the planner, which is part of Flink and not
of your JAR, turns it
+into operators again on every start. A newer Flink version can plan the same
query differently, for
+example because of new optimizer rules or a new version of an operator, and
the state in the
+savepoint then doesn't match the new operators.
+
+A [compiled plan]({{< ref "docs/concepts/sql-table-concepts/compiled_plan" >}})
+stores the result of the planning: the execution nodes of the pipeline with
their IDs and versions.
+A newer Flink version runs the plan as it is, as long as it still supports the
versions of the
+execution nodes in the plan, so that the operators find their state in the
savepoint.
+`COMPILE AND EXECUTE PLAN` compiles the plan only if the file doesn't exist
yet, and otherwise runs
+the plan from the file. Run the same statement on every Flink version:
+
+{{< tabs "upgrade-with-compiled-plan" >}}
+{{< tab "SQL CLI" >}}
+```sql
+-- 1. When you first deploy the job: create the tables and register the
functions
+-- that the query uses, then run
+COMPILE AND EXECUTE PLAN 's3://my-bucket/plans/my-job.json' FOR
+INSERT INTO sink
+SELECT * FROM MyPtf(input => TABLE src PARTITION BY user_id, uid => 'my-ptf');
+
+-- 2. Before the upgrade, stop the job with a savepoint. SHOW JOBS lists the
job ID,
+-- and STOP JOB prints the path of the savepoint.
+SET 'execution.checkpointing.savepoint-dir' = 's3://my-bucket/savepoints';
+STOP JOB '<job_id>' WITH SAVEPOINT;
+
+-- 3. On the new Flink version: create the same tables and register the same
functions,
+-- set the savepoint from step 2, and run the same statement.
+SET 'execution.state-recovery.path' =
's3://my-bucket/savepoints/savepoint-...';
+COMPILE AND EXECUTE PLAN 's3://my-bucket/plans/my-job.json' FOR
+INSERT INTO sink
+SELECT * FROM MyPtf(input => TABLE src PARTITION BY user_id, uid => 'my-ptf');
+-- The setting applies to every later job of the session.
+RESET 'execution.state-recovery.path';
+```
+{{< /tab >}}
+{{< tab "Java" >}}
+```java
+TableEnvironment tEnv =
TableEnvironment.create(EnvironmentSettings.inStreamingMode());
+tEnv.executeSql("CREATE TABLE src (...) WITH (...)");
+tEnv.executeSql("CREATE TABLE sink (...) WITH (...)");
+tEnv.createTemporarySystemFunction("MyPtf", MyPtf.class);
+
+tEnv.executeSql(
+ "COMPILE AND EXECUTE PLAN 's3://my-bucket/plans/my-job.json' FOR "
+ + "INSERT INTO sink "
+ + "SELECT * FROM MyPtf(input => TABLE src PARTITION BY
user_id, uid => 'my-ptf')");
+```
-Be aware that any change to both the query and the Flink version could lead to
state incompatibility.
-Every new major-minor Flink version (e.g. `1.12` to `1.13`) might introduce
new optimizer rules or more
-specialized runtime operators that change the execution plan. However, the
community tries to keep patch
-versions state-compatible (e.g. `1.13.1` to `1.13.2`).
+Instead of a file, the plan can also be part of the JAR of the job. Compile it
once, for example with
+`tEnv.compilePlanSql("INSERT INTO ...").writeToFile(...)`, add the file to the
resources of the
+project, and run it with:
+
+```java
+String plan;
+try (InputStream in = MyJob.class.getResourceAsStream("/plans/my-job.json")) {
+ plan = new String(in.readAllBytes(), StandardCharsets.UTF_8);
+}
Review Comment:
Would be great if you can fix it in a follow up and keep the example as is
in this PR for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]