MartijnVisser commented on code in PR #29934: URL: https://github.com/apache/flink/pull/29934#discussion_r4601298515
########## docs/content/docs/concepts/sql-table-concepts/compiled_plan.md: ########## @@ -0,0 +1,326 @@ +--- +title: "Compiled Plans" +weight: 5 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Compiled Plans + +The planner translates a Table API or SQL program into a pipeline of operators every time you +submit it. A compiled plan is the result of this translation, written to a JSON file. It contains +the execution nodes of the pipeline with their IDs and versions, the edges between them, the tables +and functions they use, and the values of the options that the nodes depend on. When you run a +plan, Flink doesn't plan the query again, so the job gets the same operators with the same IDs +every time, including on a newer Flink version within the limits in +[Flink Versions](#flink-versions). + +Use a compiled plan when you upgrade a job to a newer Flink version and need it to keep its state, +see [Upgrading the Flink Version]({{< ref "docs/ops/upgrading" >}}#upgrading-the-flink-version), and to set the state +TTL of each stateful operator, see +[Configure Operator-level State TTL]({{< ref "docs/concepts/sql-table-concepts/overview" >}}#configure-operator-level-state-ttl). + +## Compile a Plan + +The `COMPILE PLAN` statement writes the plan of an `INSERT` statement or a statement set +to a JSON file. + +- Run a `COMPILE PLAN` statement + +{{< tabs "compile-plan" >}} +{{< tab "Java" >}} +```java +TableEnvironment tableEnv = TableEnvironment.create(EnvironmentSettings.inStreamingMode()); +tableEnv.executeSql( + "CREATE TABLE orders (order_id BIGINT, order_line_id BIGINT, buyer_id BIGINT, ...)"); +tableEnv.executeSql( + "CREATE TABLE line_orders (order_line_id BIGINT, order_status TINYINT, ...)"); +tableEnv.executeSql( + "CREATE TABLE enriched_orders (order_id BIGINT, order_line_id BIGINT, order_status TINYINT, ...)"); + +// CompilePlan#writeToFile only supports a local file path, if you need to write to remote filesystem, +// please use tableEnv.executeSql("COMPILE PLAN 'hdfs://path/to/plan.json' FOR ...") +CompiledPlan compiledPlan = + tableEnv.compilePlanSql( + "INSERT INTO enriched_orders \n" + + "SELECT a.order_id, a.order_line_id, b.order_status, ... \n" + + "FROM orders a JOIN line_orders b ON a.order_line_id = b.order_line_id"); + +compiledPlan.writeToFile("/path/to/plan.json"); + +``` + +{{< /tab >}} +{{< tab "Scala" >}} Review Comment: Removed both Scala tabs. ########## docs/content/docs/concepts/sql-table-concepts/compiled_plan.md: ########## @@ -0,0 +1,326 @@ +--- +title: "Compiled Plans" +weight: 5 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Compiled Plans + +The planner translates a Table API or SQL program into a pipeline of operators every time you +submit it. A compiled plan is the result of this translation, written to a JSON file. It contains Review Comment: Done. ########## docs/content/docs/concepts/sql-table-concepts/compiled_plan.md: ########## @@ -0,0 +1,326 @@ +--- +title: "Compiled Plans" +weight: 5 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Compiled Plans + +The planner translates a Table API or SQL program into a pipeline of operators every time you +submit it. A compiled plan is the result of this translation, written to a JSON file. It contains +the execution nodes of the pipeline with their IDs and versions, the edges between them, the tables +and functions they use, and the values of the options that the nodes depend on. When you run a +plan, Flink doesn't plan the query again, so the job gets the same operators with the same IDs +every time, including on a newer Flink version within the limits in +[Flink Versions](#flink-versions). + +Use a compiled plan when you upgrade a job to a newer Flink version and need it to keep its state, Review Comment: Done. ########## docs/content/docs/concepts/sql-table-concepts/compiled_plan.md: ########## @@ -0,0 +1,326 @@ +--- +title: "Compiled Plans" +weight: 5 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Compiled Plans + +The planner translates a Table API or SQL program into a pipeline of operators every time you +submit it. A compiled plan is the result of this translation, written to a JSON file. It contains +the execution nodes of the pipeline with their IDs and versions, the edges between them, the tables +and functions they use, and the values of the options that the nodes depend on. When you run a +plan, Flink doesn't plan the query again, so the job gets the same operators with the same IDs +every time, including on a newer Flink version within the limits in +[Flink Versions](#flink-versions). + +Use a compiled plan when you upgrade a job to a newer Flink version and need it to keep its state, +see [Upgrading the Flink Version]({{< ref "docs/ops/upgrading" >}}#upgrading-the-flink-version), and to set the state +TTL of each stateful operator, see +[Configure Operator-level State TTL]({{< ref "docs/concepts/sql-table-concepts/overview" >}}#configure-operator-level-state-ttl). + +## Compile a Plan + +The `COMPILE PLAN` statement writes the plan of an `INSERT` statement or a statement set +to a JSON file. + +- Run a `COMPILE PLAN` statement + +{{< tabs "compile-plan" >}} +{{< tab "Java" >}} +```java +TableEnvironment tableEnv = TableEnvironment.create(EnvironmentSettings.inStreamingMode()); +tableEnv.executeSql( + "CREATE TABLE orders (order_id BIGINT, order_line_id BIGINT, buyer_id BIGINT, ...)"); +tableEnv.executeSql( + "CREATE TABLE line_orders (order_line_id BIGINT, order_status TINYINT, ...)"); +tableEnv.executeSql( + "CREATE TABLE enriched_orders (order_id BIGINT, order_line_id BIGINT, order_status TINYINT, ...)"); + +// CompilePlan#writeToFile only supports a local file path, if you need to write to remote filesystem, +// please use tableEnv.executeSql("COMPILE PLAN 'hdfs://path/to/plan.json' FOR ...") +CompiledPlan compiledPlan = + tableEnv.compilePlanSql( + "INSERT INTO enriched_orders \n" + + "SELECT a.order_id, a.order_line_id, b.order_status, ... \n" + + "FROM orders a JOIN line_orders b ON a.order_line_id = b.order_line_id"); + +compiledPlan.writeToFile("/path/to/plan.json"); + +``` + +{{< /tab >}} +{{< tab "Scala" >}} +```scala +val tableEnv = TableEnvironment.create(EnvironmentSettings.inStreamingMode()) +tableEnv.executeSql( + "CREATE TABLE orders (order_id BIGINT, order_line_id BIGINT, buyer_id BIGINT, ...)") +tableEnv.executeSql( + "CREATE TABLE line_orders (order_line_id BIGINT, order_status TINYINT, ...)") +tableEnv.executeSql( + "CREATE TABLE enriched_orders (order_id BIGINT, order_line_id BIGINT, order_status TINYINT, ...)") + +val compiledPlan = + tableEnv.compilePlanSql( + """ + |INSERT INTO enriched_orders + |SELECT a.order_id, a.order_line_id, b.order_status, ... + |FROM orders a JOIN line_orders b ON a.order_line_id = b.order_line_id + |""".stripMargin) +// CompilePlan#writeToFile only supports a local file path, if you need to write to remote filesystem, +// please use tableEnv.executeSql("COMPILE PLAN 'hdfs://path/to/plan.json' FOR ...") +compiledPlan.writeToFile("/path/to/plan.json") +``` +{{< /tab >}} +{{< tab "SQL CLI" >}} + +```sql +Flink SQL> CREATE TABLE orders (order_id BIGINT, order_line_id BIGINT, buyer_id BIGINT, ...); +[INFO] Execute statement succeeded. + +Flink SQL> CREATE TABLE line_orders (order_line_id BIGINT, order_status TINYINT, ...); +[INFO] Execute statement succeeded. + +Flink SQL> CREATE TABLE enriched_orders (order_id BIGINT, order_line_id BIGINT, order_status TINYINT, ...); +[INFO] Execute statement succeeded. + +Flink SQL> COMPILE PLAN 'file:///path/to/plan.json' FOR INSERT INTO enriched_orders +> SELECT a.order_id, a.order_line_id, b.order_status, ... +> FROM orders a JOIN line_orders b ON a.order_line_id = b.order_line_id; +[INFO] Execute statement succeeded. +``` + +{{< /tab >}} +{{< /tabs >}} + + +- SQL Syntax + + ```sql + COMPILE PLAN <plan_file_path> [IF NOT EXISTS] FOR <insert_statement>|<statement_set>; + + statement_set: + STATEMENT SET + BEGIN + insert_statement; + ... + insert_statement; + END; + + insert_statement: + <insert_from_select>|<insert_from_values> + ``` + This will generate a JSON file at `/path/to/plan.json`. + +{{< hint info >}} +`COMPILE PLAN` statement supports writing the plan to a remote [filesystem]({{< ref "docs/deployment/filesystems/overview" >}}) scheme like `hdfs://` or `s3://`. +Please be sure that the target path has set up the write access. +{{< /hint >}} + +If the file already exists, `COMPILE PLAN` fails: + +```text +org.apache.flink.table.api.TableException: Cannot overwrite the plan file 'file:///path/to/plan.json'. Either manually remove the file or, if you're debugging your job, set the option 'table.plan.force-recompile' to true. +``` + +With `IF NOT EXISTS`, `COMPILE PLAN` keeps the existing file and doesn't compile the statement. +`CompiledPlan#writeToFile(path, true)` also keeps an existing file. With +`table.plan.force-recompile` = `true`, `COMPILE PLAN` overwrites the file. Only use this while you +develop a job: a new compilation can assign other operator IDs than the ones in a savepoint. + +The SQL Client or SQL Gateway that runs the statement reads and writes the plan file, so it needs +access to the path. With a remote SQL Gateway, a local path is a path on the machine of the +Gateway. + +## Execute a Plan + +`EXECUTE PLAN` statement will deserialize the specified file back to execution plan of the current table program and then submit the job. + +- Run an `EXECUTE PLAN` statement +{{< tabs "execute-plan" >}} +{{< tab "Java" >}} +```java +TableEnvironment tableEnv = TableEnvironment.create(EnvironmentSettings.inStreamingMode()); +tableEnv.executeSql( + "CREATE TABLE orders (order_id BIGINT, order_line_id BIGINT, buyer_id BIGINT, ...)"); +tableEnv.executeSql( + "CREATE TABLE line_orders (order_line_id BIGINT, order_status TINYINT, ...)"); +tableEnv.executeSql( + "CREATE TABLE enriched_orders (order_id BIGINT, order_line_id BIGINT, order_status TINYINT, ...)"); + +// PlanReference#fromFile only supports a local file path, if you need to read from remote filesystem, +// please use tableEnv.executeSql("EXECUTE PLAN 'hdfs://path/to/plan.json'").await(); +tableEnv.loadPlan(PlanReference.fromFile("/path/to/plan.json")).execute().await(); + +``` + +{{< /tab >}} +{{< tab "Scala" >}} +```scala +val tableEnv = TableEnvironment.create(EnvironmentSettings.inStreamingMode()) +tableEnv.executeSql( + "CREATE TABLE orders (order_id BIGINT, order_line_id BIGINT, buyer_id BIGINT, ...)") +tableEnv.executeSql( + "CREATE TABLE line_orders (order_line_id BIGINT, order_status TINYINT, ...)") +tableEnv.executeSql( + "CREATE TABLE enriched_orders (order_id BIGINT, order_line_id BIGINT, order_status TINYINT, ...)") + +// PlanReference#fromFile only supports a local file path, if you need to read from remote filesystem, +// please use tableEnv.executeSql("EXECUTE PLAN 'hdfs://path/to/plan.json'").await() +tableEnv.loadPlan(PlanReference.fromFile("/path/to/plan.json")).execute().await() +``` +{{< /tab >}} +{{< tab "SQL CLI" >}} + +```sql +Flink SQL> CREATE TABLE orders (order_id BIGINT, order_line_id BIGINT, buyer_id BIGINT, ...); +[INFO] Execute statement succeeded. + +Flink SQL> CREATE TABLE line_orders (order_line_id BIGINT, order_status TINYINT, ...); +[INFO] Execute statement succeeded. + +Flink SQL> CREATE TABLE enriched_orders (order_id BIGINT, order_line_id BIGINT, order_status TINYINT, ...); +[INFO] Execute statement succeeded. + +Flink SQL> EXECUTE PLAN 'file:///path/to/plan.json'; +[INFO] Submitting SQL update statement to the cluster... +[INFO] SQL update statement has been successfully submitted to the cluster: +Job ID: 79fbe3fa497e4689165dd81b1d225ea8 +``` + +{{< /tab >}} +{{< /tabs >}} + +- SQL Syntax + + ```sql + EXECUTE PLAN <plan_file_path>; + ``` + This will deserialize the JSON file and submit an insert statement job. + +To run a plan that your Table API program ships as a resource in its JAR, read the resource +and use `PlanReference.fromJsonString`, as in the Java example in +[Upgrading the Flink Version]({{< ref "docs/ops/upgrading" >}}#upgrading-the-flink-version). +`PlanReference.fromResource` fails for a file inside a JAR. + +### Tables and Functions + +With the default `table.plan.compile.catalog-objects` = `ALL`, a plan stores each catalog table +with its schema and options. `EXECUTE PLAN` then also works in a new session in which the tables +haven't been created. If a table is in the catalog, a restore takes the options that its connector +allows to change from there, such as the `path` of the filesystem connector. See +`table.plan.restore.catalog-objects`. + +Temporary tables and functions are stored only by their name. Create them again before you run +the plan, otherwise `EXECUTE PLAN` fails: + +```text +org.apache.flink.table.api.TableException: Cannot resolve table 'default_catalog.default_database.orders' and the persisted plan does not include all required catalog table metadata. Make sure a registered catalog contains the table when restoring or the table is available as a temporary table. Otherwise regenerate the plan with 'table.plan.compile.catalog-objects' != 'IDENTIFIER' and make sure the table was not compiled as a temporary table. +``` + +The plan assumes that its tables and functions don't change. If a table in the catalog has another +schema than in the plan, `EXECUTE PLAN` fails with `The schema of table '...' from the persisted plan +does not match the schema loaded from the catalog`. A change of the signature of a function isn't +supported either. + +### Options Review Comment: Added, plus a sentence that only the options stored in the plan are taken from the plan, to match the `not-null-enforcer` example below. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
