twalthr commented on code in PR #24155:
URL: https://github.com/apache/flink/pull/24155#discussion_r1471452167
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/TableDescriptor.java:
##########
@@ -355,6 +355,22 @@ public Builder distributedBy(TableDistribution
distribution) {
return this;
}
+ public Builder distributedByHash(String... bucketKeys) {
+ return distributedByHash(null, bucketKeys);
+ }
+
+ public Builder distributedByHash(Integer number, String... bucketKeys)
{
+ return
distributedBy(TableDistribution.ofHash(Arrays.asList(bucketKeys), number));
+ }
+
+ public Builder distributedByRange(String... bucketKeys) {
+ return distributedByRange(null, bucketKeys);
+ }
+
+ public Builder distributedByRange(Integer number, String...
bucketKeys) {
+ return
distributedBy(TableDistribution.ofRange(Arrays.asList(bucketKeys), number));
+ }
Review Comment:
add distributedInto(int number) for the UNKNOWN case
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/SqlCreateTableConverter.java:
##########
@@ -244,7 +244,10 @@ private Optional<CatalogTable.TableDistribution>
mergeDistribution(
if (sqlCreateTable.getDistribution() != null) {
CatalogTable.TableDistribution.Kind kind =
CatalogTable.TableDistribution.Kind.valueOf(
-
sqlCreateTable.getDistribution().getDistributionKind());
+ sqlCreateTable
+ .getDistribution()
+ .getDistributionKind()
+ .orElse("UNKNOWN"));
Review Comment:
`Kind.UNKNOWN.toString()` to allow for references of this constant.
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/TableDescriptor.java:
##########
@@ -355,6 +355,22 @@ public Builder distributedBy(TableDistribution
distribution) {
return this;
}
Review Comment:
we can drop this method
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/TableDistribution.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.table.utils.EncodingUtils;
+
+import javax.annotation.Nullable;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/** Distribution specification. */
+@PublicEvolving
+public class TableDistribution {
+
+ private final Kind kind;
+ private final @Nullable Integer bucketCount;
+ private final List<String> bucketKeys;
+
+ @PublicEvolving
+ public TableDistribution(Kind kind, @Nullable Integer bucketCount,
List<String> bucketKeys) {
Review Comment:
remove @PublicEvolving as the class level one covers it already
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/TableDistribution.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.table.utils.EncodingUtils;
+
+import javax.annotation.Nullable;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/** Distribution specification. */
+@PublicEvolving
+public class TableDistribution {
+
+ private final Kind kind;
+ private final @Nullable Integer bucketCount;
+ private final List<String> bucketKeys;
+
+ @PublicEvolving
+ public TableDistribution(Kind kind, @Nullable Integer bucketCount,
List<String> bucketKeys) {
+ this.kind = kind;
+ this.bucketCount = bucketCount;
+ this.bucketKeys = bucketKeys;
+ }
+
+ /** Connector-dependent distribution with a declared number of buckets. */
+ public static TableDistribution ofUnknown(int bucketCount) {
+ return new TableDistribution(Kind.UNKNOWN, bucketCount,
Collections.emptyList());
+ }
+
+ /** Hash distribution over on the given keys among the declared number of
buckets. */
Review Comment:
is `over on` proper English?
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/TableDistribution.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.table.utils.EncodingUtils;
+
+import javax.annotation.Nullable;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/** Distribution specification. */
+@PublicEvolving
+public class TableDistribution {
+
+ private final Kind kind;
+ private final @Nullable Integer bucketCount;
+ private final List<String> bucketKeys;
+
+ @PublicEvolving
+ public TableDistribution(Kind kind, @Nullable Integer bucketCount,
List<String> bucketKeys) {
+ this.kind = kind;
+ this.bucketCount = bucketCount;
+ this.bucketKeys = bucketKeys;
+ }
+
+ /** Connector-dependent distribution with a declared number of buckets. */
+ public static TableDistribution ofUnknown(int bucketCount) {
+ return new TableDistribution(Kind.UNKNOWN, bucketCount,
Collections.emptyList());
+ }
+
+ /** Hash distribution over on the given keys among the declared number of
buckets. */
+ public static TableDistribution ofHash(List<String> bucketKeys, @Nullable
Integer bucketCount) {
+ return new TableDistribution(Kind.HASH, bucketCount, bucketKeys);
+ }
+
+ /** Range distribution over on the given keys among the declared number of
buckets. */
+ public static TableDistribution ofRange(
+ List<String> bucketKeys, @Nullable Integer bucketCount) {
+ return new TableDistribution(Kind.RANGE, bucketCount, bucketKeys);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ TableDistribution that = (TableDistribution) o;
+ return kind == that.kind
+ && Objects.equals(bucketCount, that.bucketCount)
+ && Objects.equals(bucketKeys, that.bucketKeys);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(kind, bucketCount, bucketKeys);
+ }
+
+ /** Distribution kind. */
+ @PublicEvolving
Review Comment:
can be removed as covered by class level annotation
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/CatalogTable.java:
##########
@@ -82,7 +85,29 @@ static CatalogTable of(
List<String> partitionKeys,
Map<String, String> options,
@Nullable Long snapshot) {
- return new DefaultCatalogTable(schema, comment, partitionKeys,
options, snapshot);
+ return new DefaultCatalogTable(
+ schema, comment, partitionKeys, options, snapshot,
Optional.empty());
+ }
+
+ /**
+ * Creates an instance of {@link CatalogTable} with a specific snapshot.
+ *
+ * @param schema unresolved schema
+ * @param comment optional comment
+ * @param distribution distribution of the table
+ * @param partitionKeys list of partition keys or an empty list if not
partitioned
+ * @param options options to configure the connector
+ * @param snapshot table snapshot of the table
+ */
+ static CatalogTable of(
Review Comment:
Yes, hat would be great. We should not introduce a `of` method that we want
to deprecate in the next version again. No need to remove all existing usages
of the deprecated method. This can be done in a followup ticket.
You can checkout out `org.apache.flink.table.types.inference.TypeInference`
and follow the pattern for a builder there. Meaning `CatalogTable.newBuilder()`
with a static inner class `Builder`.
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/TableDescriptor.java:
##########
@@ -334,6 +352,12 @@ public Builder format(
return this;
}
+ /** Define which columns this table is distributed by. */
+ public Builder distributedBy(CatalogTable.TableDistribution
tableDistribution) {
Review Comment:
yes, but just `distributedInto` as in SQL
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/abilities/sink/BucketingSpec.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.abilities.sink;
+
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+
+import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonTypeName;
+
+/**
+ * No properties. This only checks whether the interface is implemented again
during deserialization
+ */
+@JsonIgnoreProperties(ignoreUnknown = true)
+@JsonTypeName("Bucketing")
Review Comment:
> Are you referring to @JsonTypeName("Bucketing") or something else?
Yes, I also don't recall why I added it. Might be a mistake or feedback that
I got during ML discussion. In any case we should only add code if we know what
it is doing (and we want it that way).
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/TableDistribution.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.table.utils.EncodingUtils;
+
+import javax.annotation.Nullable;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/** Distribution specification. */
+@PublicEvolving
+public class TableDistribution {
+
+ private final Kind kind;
+ private final @Nullable Integer bucketCount;
+ private final List<String> bucketKeys;
+
+ @PublicEvolving
+ public TableDistribution(Kind kind, @Nullable Integer bucketCount,
List<String> bucketKeys) {
Review Comment:
can be private
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/TableDescriptor.java:
##########
@@ -355,6 +355,22 @@ public Builder distributedBy(TableDistribution
distribution) {
return this;
}
+ public Builder distributedByHash(String... bucketKeys) {
+ return distributedByHash(null, bucketKeys);
+ }
+
+ public Builder distributedByHash(Integer number, String... bucketKeys)
{
Review Comment:
make `number` `int` here and below to avoid NPEs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
