AHeise commented on code in PR #29936:
URL: https://github.com/apache/flink/pull/29936#discussion_r4597814348


##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/runtime/stream/sql/MaterializedTableTableApiITCase.java:
##########
@@ -856,20 +884,12 @@ void 
testRedeployWithDroppedColumnIsRejectedAndLeavesRefreshJobSuspended(
     @Test
     void testRedeployMaterializedTableWithMaterializeInto(@TempDir Path 
temporaryPath)
             throws Exception {
-        tEnv.executeSql(
-                "CREATE MATERIALIZED TABLE users_shops"
-                        + " PARTITIONED BY (ds)\n"
-                        + UPDATING_VALUES_SINK
-                        + " FRESHNESS = INTERVAL '30' SECOND\n"
-                        + " AS SELECT \n"
-                        + "  user_id,\n"
-                        + "  shop_id,\n"
-                        + "  ds,\n"
-                        + "  SUM (payment_amount_cents) AS payed_buy_fee_sum\n"
-                        + " FROM (\n"
-                        + "    SELECT user_id, shop_id, 
DATE_FORMAT(order_created_at, 'yyyy-MM-dd') AS ds, payment_amount_cents FROM 
datagenSource"
-                        + " ) AS tmp\n"
-                        + " GROUP BY (user_id, shop_id, ds)");
+        usersShopsQuery()
+                .dropColumns($("pv"))
+                .materializeInto(
+                        "users_shops",
+                        
usersShopsDescriptor().freshness(Duration.ofSeconds(30)).build())

Review Comment:
   All redeploys in this class go through `materializeInto(path[, Duration])`. 
So nothing checks that a declared comment is ignored on redeploy, which holds 
only because the executor rebuilds the alter from table changes. Could the 
redeploy below use a descriptor with a comment and a subset of the options, and 
assert that the comment is unchanged and the options equal the declared ones?
   
   A small `MaterializedTableDescriptorTest` for 
`toString`/`equals`/`toBuilder`, like `TableDescriptorTest`, would be nice too.



##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/MaterializedPipeline.java:
##########
@@ -53,9 +53,13 @@ public interface MaterializedPipeline {
      * in full refresh mode is altered without changing its refresh workflow.
      *
      * <p>An existing materialized table keeps its freshness, comment and 
partition keys, so a
-     * freshness given to {@link Table#materializeInto(String, 
java.time.Duration)} is ignored. Its
-     * options are reset and its distribution is dropped, as by a {@code 
CREATE OR ALTER
-     * MATERIALIZED TABLE} without {@code WITH} and {@code DISTRIBUTED BY} 
clauses.
+     * declared one is ignored. A declared {@link 
RefreshStrategy#continuous()} or {@link

Review Comment:
   The rewrite dropped the sentence about options and distribution, so the 
Javadoc now lists what's kept but not what's replaced. Could we add something 
like "Its options and distribution are replaced by the declared ones, so 
undeclared options are removed and the distribution is dropped if none is 
declared"? The docs page says this, but for a `@PublicEvolving` type the 
Javadoc is the contract.
   
   A short note on `Builder#option`/`distributedBy*` would also help, mirroring 
the "only applied when created" note on `partitionedBy`/`comment`/`freshness`. 
Right now that note suggests the others are kept as well.



##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/materializedtable/DescriptorMaterializedTableMergeContext.java:
##########
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.api.internal.materializedtable;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.MaterializedTableDescriptor;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Schema.UnresolvedColumn;
+import org.apache.flink.table.api.Schema.UnresolvedMetadataColumn;
+import org.apache.flink.table.api.Schema.UnresolvedPhysicalColumn;
+import org.apache.flink.table.api.Schema.UnresolvedWatermarkSpec;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.catalog.CatalogManager;
+import 
org.apache.flink.table.catalog.CatalogMaterializedTable.LogicalRefreshMode;
+import org.apache.flink.table.catalog.Column;
+import org.apache.flink.table.catalog.DataTypeFactory;
+import org.apache.flink.table.catalog.IntervalFreshness;
+import org.apache.flink.table.catalog.ResolvedCatalogMaterializedTable;
+import org.apache.flink.table.catalog.ResolvedSchema;
+import org.apache.flink.table.catalog.StartMode;
+import org.apache.flink.table.catalog.TableDistribution;
+import org.apache.flink.table.operations.QueryOperation;
+import 
org.apache.flink.table.operations.materializedtable.MaterializedTableMergeContext;
+import org.apache.flink.table.operations.utils.AsQuerySchemaMerger;
+import org.apache.flink.table.types.AbstractDataType;
+import org.apache.flink.table.types.DataType;
+
+import javax.annotation.Nullable;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static 
org.apache.flink.table.operations.materializedtable.MaterializedTableSchemaUtils.METADATA_PERSISTED_COLUMN_KIND;
+import static 
org.apache.flink.table.operations.materializedtable.MaterializedTableSchemaUtils.PHYSICAL_COLUMN_KIND;
+import static 
org.apache.flink.table.operations.materializedtable.MaterializedTableSchemaUtils.checkPersistedColumnProducedByQuery;
+import static 
org.apache.flink.table.types.utils.TypeConversions.fromLogicalToDataType;
+
+/**
+ * The {@link MaterializedTableMergeContext} of {@code Table#materializeInto}, 
with the semantics of
+ * the equivalent {@code CREATE [OR ALTER] MATERIALIZED TABLE} statement.
+ */
+@Internal
+final class DescriptorMaterializedTableMergeContext implements 
MaterializedTableMergeContext {
+
+    private static final String OPERATION_NAME = "CREATE OR ALTER MATERIALIZED 
TABLE";
+
+    private final MaterializedTableDescriptor descriptor;
+    private final QueryOperation queryOperation;
+    private final String serializedQuery;
+    private final DataTypeFactory dataTypeFactory;
+    private final ResolvedSchema querySchema;
+
+    DescriptorMaterializedTableMergeContext(
+            MaterializedTableDescriptor descriptor,
+            QueryOperation queryOperation,
+            String serializedQuery,
+            CatalogManager catalogManager) {
+        this.descriptor = descriptor;
+        this.queryOperation = queryOperation;
+        this.serializedQuery = serializedQuery;
+        this.dataTypeFactory = catalogManager.getDataTypeFactory();
+        this.querySchema = 
toPhysicalSchema(queryOperation.getResolvedSchema());
+    }
+
+    @Override
+    public boolean hasSchemaDefinition() {
+        // A column list defines the schema only if it starts with a physical 
column.
+        return descriptor
+                .getSchema()
+                .map(
+                        schema ->
+                                !schema.getColumns().isEmpty()
+                                        && schema.getColumns().get(0)
+                                                instanceof 
UnresolvedPhysicalColumn)
+                .orElse(false);
+    }
+
+    @Override
+    public boolean hasConstraintDefinition() {
+        return 
descriptor.getSchema().flatMap(Schema::getPrimaryKey).isPresent()
+                || hasSchemaDefinition();
+    }
+
+    @Override
+    public Schema getMergedSchema() {
+        final Schema querySchemaDeclaration =
+                Schema.newBuilder().fromResolvedSchema(querySchema).build();
+        final Schema declaredSchema = descriptor.getSchema().orElse(null);
+        if (declaredSchema == null) {
+            return querySchemaDeclaration;
+        }
+        rejectUnsupportedSchemaParts(declaredSchema);
+        checkPersistedColumnsProducedByQuery(declaredSchema);
+
+        final AsQuerySchemaMerger merger =
+                new AsQuerySchemaMerger(querySchemaDeclaration.getColumns(), 
dataTypeFactory);
+        final List<UnresolvedColumn> declaredColumns = 
declaredSchema.getColumns();
+        // Schema.Builder accepts repeated names, and the merger only rejects 
a repeat of a column
+        // the query doesn't produce.
+        final Set<String> declaredNames = new HashSet<>();
+        for (int position = 0; position < declaredColumns.size(); position++) {
+            final UnresolvedColumn column = declaredColumns.get(position);
+            if (!declaredNames.add(column.getName())) {
+                throw new ValidationException(
+                        String.format(
+                                "A column named '%s' already exists in the 
schema.",
+                                column.getName()));
+            }
+            merger.addDeclaredColumn(withDefaultConversionClass(column), 
position);
+        }
+
+        final Schema.Builder mergedSchema =
+                Schema.newBuilder().fromColumns(merger.getMergedColumns());
+        for (UnresolvedWatermarkSpec watermark : 
declaredSchema.getWatermarkSpecs()) {
+            mergedSchema.watermark(watermark.getColumnName(), 
watermark.getWatermarkExpression());
+        }
+        declaredSchema
+                .getPrimaryKey()
+                .ifPresent(
+                        primaryKey ->
+                                mergedSchema.primaryKeyNamed(

Review Comment:
   Small parity question. In SQL, `CREATE [OR ALTER] MATERIALIZED TABLE t (id 
BIGINT, PRIMARY KEY (id) NOT ENFORCED) AS ...` works because 
`SqlConstraintValidator` makes declared PK columns NOT NULL. Here, 
`Schema.newBuilder().column("id", DataTypes.BIGINT()).primaryKey("id")` fails 
with "Column 'id' is nullable", even when the query produces `id` as NOT NULL. 
That matches how `Schema`/`createTable` treat declared columns, so I'm fine 
either way.
   
   Would you consider making declared PK physical columns NOT NULL before they 
go to `AsQuerySchemaMerger`? The cast check would still reject a nullable query 
column. Otherwise, could the docs say that re-declared PK columns need 
`.notNull()`? The current sentence only mentions the query.



##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/RefreshStrategy.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.api;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/** Declares how the refresh pipeline of a materialized table runs. */
+@PublicEvolving
+public class RefreshStrategy {
+
+    /** The kind of a {@link RefreshStrategy}. */
+    @PublicEvolving
+    public enum Kind {
+        AUTOMATIC,
+        CONTINUOUS,
+        FULL
+    }
+
+    private static final RefreshStrategy AUTOMATIC = new 
RefreshStrategy(Kind.AUTOMATIC);
+    private static final RefreshStrategy CONTINUOUS = new 
RefreshStrategy(Kind.CONTINUOUS);
+    private static final RefreshStrategy FULL = new RefreshStrategy(Kind.FULL);
+
+    private final Kind kind;
+
+    protected RefreshStrategy(Kind kind) {

Review Comment:
   FLIP-616 doesn't list a constructor for `RefreshStrategy`, and nothing 
outside the class calls it. With getClass-based `equals`, a subclass (`new 
RefreshStrategy(Kind.FULL) {}`) wouldn't equal `full()` while behaving 
identically. Could the constructor be private, like 
`StartMode`/`TableDistribution`, so the factories are the only entry point? 
Widening it later is compatible; narrowing it isn't.



##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/RefreshStrategy.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.api;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/** Declares how the refresh pipeline of a materialized table runs. */
+@PublicEvolving
+public class RefreshStrategy {

Review Comment:
   Do we need a new type here? Its kinds are 1:1 with 
`CatalogMaterializedTable.LogicalRefreshMode` (`@PublicEvolving`, same 
AUTOMATIC-by-freshness semantics), and the merge context maps them by name. The 
FLIP motivates it with "more fluent declaration options", but none are defined 
yet, and the builder's `continuousRefresh()`/`fullRefresh()` already give the 
fluency. Using `LogicalRefreshMode` directly would drop a public class and the 
name mapping, and a parameterized strategy could still be added later as an 
overload.



##########
docs/content/docs/sql/materialized-table/table-api.md:
##########
@@ -46,9 +46,37 @@ MaterializedTable revenue = tEnv.from("orders")
 The schema is derived from the stored SQL text of the query, and the path is 
resolved against the
 current catalog and database. The freshness is optional and defaults to
 [materialized-table.default-freshness.continuous]({{< ref 
"docs/dev/table/config" >}}#materialized-table-default-freshness-continuous).
-The refresh mode is derived from the freshness (see [Refresh Mode]({{< ref 
"docs/sql/materialized-table/overview" >}}#refresh-mode)).
+The refresh mode is derived from the freshness, unless a 
[`MaterializedTableDescriptor`](#declaring-a-materialized-table) declares it 
(see [Refresh Mode]({{< ref "docs/sql/materialized-table/overview" 
>}}#refresh-mode)).

Review Comment:
   `#declaring-a-materialized-table` doesn't resolve, since no heading produces 
that id (same in content.zh). Maybe add a `## Declaring a Materialized Table` 
heading above the descriptor example?
   
   Nearby, line 46 still says the schema is derived from the stored SQL text. 
It could mention that a declared schema is merged onto it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to