diqiu50 commented on code in PR #14197:
URL: https://github.com/apache/gravitino/pull/14197#discussion_r4384630632
##########
spark-connector/spark-common/src/main/spark35/org/apache/gravitino/spark/connector/paimon/GravitinoPaimonCatalog.java:
##########
@@ -107,4 +231,103 @@ public Procedure loadProcedure(Identifier identifier)
throws NoSuchProcedureExce
}
throw new NoSuchProcedureException(identifier);
}
+
+ private static String[] parseColumns(
+ String option, Map<String, String> properties, StructType schema) {
+ String value = properties.get(option);
+ Preconditions.checkArgument(StringUtils.isNotBlank(value), "%s must not be
empty.", option);
+ String[] columns = Arrays.stream(value.split(",",
-1)).map(String::trim).toArray(String[]::new);
+ validateColumns(option, columns, schema);
+ return columns;
+ }
+
+ private static void validateColumns(String option, String[] columns,
StructType schema) {
Review Comment:
Column names are matched case-sensitively against `schema.fieldNames()`, but
Spark is case-insensitive by default, so `'primary-key' = 'ID'` with a column
`id` would be rejected. Should we respect `spark.sql.caseSensitive` or resolve
the names to the schema's actual names?
##########
spark-connector/spark-common/src/main/spark35/org/apache/gravitino/spark/connector/paimon/SparkPaimonTable.java:
##########
@@ -69,7 +78,33 @@ public StructType schema() {
@Override
public Map<String, String> properties() {
- return gravitinoTableInfoHelper.properties();
+ Map<String, String> properties = gravitinoTableInfoHelper.properties();
+ Index[] indexes = gravitinoTable.index();
+ if (indexes != null) {
+ Arrays.stream(indexes)
+ .filter(index -> index.type() == Index.IndexType.PRIMARY_KEY)
+ .findFirst()
+ .ifPresent(
+ index ->
+ properties.put(
+ PaimonConstants.PRIMARY_KEY,
+ Arrays.stream(index.fieldNames())
+ .map(fieldName -> String.join(".", fieldName))
+ .collect(Collectors.joining(","))));
+ }
+ Distribution distribution = gravitinoTable.distribution();
+ if (!Distributions.isNone(distribution)) {
+ properties.put(PaimonConstants.BUCKET_NUM,
String.valueOf(distribution.number()));
+ properties.remove(PaimonConstants.BUCKET_KEY);
+ if (distribution.expressions() != null &&
distribution.expressions().length > 0) {
+ properties.put(
+ PaimonConstants.BUCKET_KEY,
+ Arrays.stream(distribution.expressions())
+ .map(expression -> ((NamedReference.FieldReference)
expression).fieldName()[0])
Review Comment:
This unchecked cast will throw `ClassCastException` if the distribution
expression is not a `FieldReference`. Could we add an `instanceof` check or a
`Preconditions` check with a clear message?
##########
spark-connector/spark-common/src/main/spark35/org/apache/gravitino/spark/connector/paimon/GravitinoPaimonCatalog.java:
##########
@@ -107,4 +231,103 @@ public Procedure loadProcedure(Identifier identifier)
throws NoSuchProcedureExce
}
throw new NoSuchProcedureException(identifier);
}
+
+ private static String[] parseColumns(
+ String option, Map<String, String> properties, StructType schema) {
+ String value = properties.get(option);
+ Preconditions.checkArgument(StringUtils.isNotBlank(value), "%s must not be
empty.", option);
+ String[] columns = Arrays.stream(value.split(",",
-1)).map(String::trim).toArray(String[]::new);
+ validateColumns(option, columns, schema);
+ return columns;
+ }
+
+ private static void validateColumns(String option, String[] columns,
StructType schema) {
+ Set<String> schemaColumns = new
HashSet<>(Arrays.asList(schema.fieldNames()));
+ Set<String> seen = new HashSet<>();
+ for (String column : columns) {
+ Preconditions.checkArgument(
+ !column.isEmpty(), "%s must contain non-empty, top-level column
names.", option);
+ Preconditions.checkArgument(
+ schemaColumns.contains(column), "Column %s in %s does not exist.",
column, option);
+ Preconditions.checkArgument(seen.add(column), "Duplicate column %s in
%s.", column, option);
+ }
+ }
+
+ private static Distribution toDistribution(
Review Comment:
Most of these checks (bucket keys vs. partition columns, PK must include
partition columns for fixed buckets, append table bucket keys, etc.) duplicate
Paimon's own schema validation, which already runs on the server side when the
table is created. Could we keep only the conversion and conflict checks (e.g.
`CLUSTERED BY` vs. table properties) here and leave the semantic validation to
Paimon? Otherwise these rules may drift from Paimon over time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
