JingsongLi commented on code in PR #10978:
URL: https://github.com/apache/paimon/pull/10978#discussion_r4478098274
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java:
##########
@@ -437,6 +439,52 @@ public static void validateTableSchema(TableSchema schema,
Set<String> dynamicOp
validatePkClusteringOverride(options);
validateManifestSort(schema, options);
+
+ validateExternalPaths(options);
+ }
+
+ private static void validateExternalPaths(CoreOptions options) {
+ String externalPaths = options.dataFileExternalPaths();
+ if (externalPaths == null || externalPaths.isEmpty()) {
+ return;
+ }
+
+ boolean templateStrategy =
+ options.externalPathStrategy() ==
ExternalPathStrategy.ENTROPY_INJECT_TEMPLATE;
+ String[] paths = externalPaths.split(",");
+
+ if (templateStrategy && paths.length > 1) {
+ throw new IllegalArgumentException(
+ String.format(
+ "'%s' only accepts a single path when '%s' is
'%s', but got %s.",
+ CoreOptions.DATA_FILE_EXTERNAL_PATHS.key(),
+
CoreOptions.DATA_FILE_EXTERNAL_PATHS_STRATEGY.key(),
+ ExternalPathStrategy.ENTROPY_INJECT_TEMPLATE,
+ paths.length));
+ }
+
+ for (String externalPath : paths) {
+ boolean isTemplate =
TemplateEntropyInjectExternalPathProvider.isTemplate(externalPath);
+ if (templateStrategy && !isTemplate) {
+ throw new IllegalArgumentException(
+ String.format(
+ "External path '%s' must contain the '%s'
placeholder when '%s' is '%s'.",
+ externalPath.trim(),
+
TemplateEntropyInjectExternalPathProvider.HASH_PLACEHOLDER,
+
CoreOptions.DATA_FILE_EXTERNAL_PATHS_STRATEGY.key(),
+ ExternalPathStrategy.ENTROPY_INJECT_TEMPLATE));
+ }
+ if (!templateStrategy && isTemplate) {
Review Comment:
[P2] Preserve literal `{hash}` directories for existing strategies
An existing `round-robin` table can use a directory literally named
`{hash}`. I created and wrote such a table with the baseline validation, then
reopened it on this head. Both scan planning and writer creation fail at this
check because runtime validation also applies to persisted schemas. Please
interpret `{hash}` only when `entropy-inject-template` is selected.
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java:
##########
@@ -437,6 +439,52 @@ public static void validateTableSchema(TableSchema schema,
Set<String> dynamicOp
validatePkClusteringOverride(options);
validateManifestSort(schema, options);
+
+ validateExternalPaths(options);
+ }
+
+ private static void validateExternalPaths(CoreOptions options) {
+ String externalPaths = options.dataFileExternalPaths();
+ if (externalPaths == null || externalPaths.isEmpty()) {
+ return;
+ }
+
+ boolean templateStrategy =
+ options.externalPathStrategy() ==
ExternalPathStrategy.ENTROPY_INJECT_TEMPLATE;
+ String[] paths = externalPaths.split(",");
+
+ if (templateStrategy && paths.length > 1) {
+ throw new IllegalArgumentException(
+ String.format(
+ "'%s' only accepts a single path when '%s' is
'%s', but got %s.",
+ CoreOptions.DATA_FILE_EXTERNAL_PATHS.key(),
+
CoreOptions.DATA_FILE_EXTERNAL_PATHS_STRATEGY.key(),
+ ExternalPathStrategy.ENTROPY_INJECT_TEMPLATE,
+ paths.length));
+ }
+
+ for (String externalPath : paths) {
+ boolean isTemplate =
TemplateEntropyInjectExternalPathProvider.isTemplate(externalPath);
Review Comment:
[P0] Keep template paths out of recursive DROP cleanup
An empty table can be created with `file:///shared/{hash}/..` because this
check sees `{hash}` in the raw option. `Path` normalizes it to
`file:///shared`, and `getSchemaExternalPaths` passes that directory to
recursive cleanup. I reproduced this with `FileSystemCatalog`. Dropping the
empty table deleted a neighboring table's committed data file while leaving
that table registered. Please validate that the placeholder survives URI path
normalization and exclude template data paths from recursive deletion.
##########
paimon-core/src/main/java/org/apache/paimon/catalog/AbstractCatalog.java:
##########
@@ -392,9 +393,24 @@ public void dropTable(Identifier identifier, boolean
ignoreIfNotExists)
FileIO tableFileIO = fileIO(tablePath);
SchemaManager schemaManager = new
FileSystemSchemaManager(tableFileIO, tablePath);
externalPaths.addAll(getSchemaExternalPaths(schemaManager.listAll()));
+ FileStoreTable table = (FileStoreTable) getTable(identifier);
+ boolean isExternal =
loadTableMetadata(identifier).isExternal();
+ Optional<TableSchema> latestMainSchema =
schemaManager.latest();
+ if (!isExternal
+ && latestMainSchema.isPresent()
+ &&
EntropyInjectTemplateFilesDeletion.isTemplate(latestMainSchema.get())) {
+ EntropyInjectTemplateFilesDeletion.delete(table);
+ }
for (String branch :
FileSystemBranchManager.branches(tableFileIO, tablePath)) {
Review Comment:
[P2] Retain cleanup references when dropping a template branch
I wrote one file only to `dev` on a template append table, dropped `dev`,
and then dropped the table. The file remained in the shared prefix.
`dropBranch` removes the metadata that points to the branch's manifests, so
this loop can no longer find the file. The orphan cleaner also rejects template
tables. Please retain the branch's cleanup references until its files can be
reclaimed, while protecting files still referenced by surviving branches.
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java:
##########
@@ -437,6 +439,52 @@ public static void validateTableSchema(TableSchema schema,
Set<String> dynamicOp
validatePkClusteringOverride(options);
validateManifestSort(schema, options);
+
+ validateExternalPaths(options);
+ }
+
+ private static void validateExternalPaths(CoreOptions options) {
+ String externalPaths = options.dataFileExternalPaths();
+ if (externalPaths == null || externalPaths.isEmpty()) {
Review Comment:
[P2] Require a non-empty template when the strategy is selected
`CREATE TABLE` accepts `entropy-inject-template` with no
`data-file.external-paths`. Writes then go to the table directory, while the
orphan cleaner still refuses the table. An explicitly empty path also makes
`DROP TABLE` fail when it constructs a `Path`. Please require exactly one
non-empty template for this strategy during CREATE and ALTER.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]