sundapeng opened a new pull request, #8878: URL: https://github.com/apache/paimon/pull/8878
### Purpose A value-only format table (`format-table.partition-path-only-value=true`) writes the null partition as a bare `__DEFAULT_PARTITION__` directory, so the partition value itself carries the `_` that the generic hidden-directory rule rejects. `PartitionPathUtils.isHiddenFile` already knows this and spares the directory whose name equals the table's default partition name — but only when it is told what that name is. `FormatTablePartitionRepair` called the five-argument `searchPartSpecAndPaths`, which passes `defaultPartValue` as null, so the rescue never applied and MSCK never saw the directory. Two ways to lose data followed: - `MSCK REPAIR TABLE ... ADD PARTITIONS` never registered the null partition; - `MSCK REPAIR TABLE ... SYNC PARTITIONS` read it as *registered but the directory is gone* and **unregistered a partition that still holds data**. Later queries then skip it without a word. `listStatusRecursively` applies the same rule while descending, so a null value on a non-leaf level hid the whole subtree rather than a single directory. ### Tests `FormatTablePartitionRepairTest`, 18 cases green. New cases cover: | case | pins | |---|---| | `repairAddsTheNullPartitionDirectoryInValueOnlyLayout` | ADD registers it (2 instead of 1 before the fix) | | `repairKeepsTheRegisteredNullPartitionInValueOnlyLayout` | SYNC is a no-op instead of unregistering live data | | `repairDescendsIntoANullPartitionSubtreeInValueOnlyLayout` | a null value on a non-leaf level no longer hides the subtree | | `repairReadsTheDefaultPartitionNameFromTableOptions` | the name comes from `partition.default-name`; hardcoding the literal passes every other case | | `repairKeepsAnUnderscoreValueInTheKeyValueLayout` | key=value layout is unaffected — the underscore is not the first character of `dt=_abc` | `CatalogManagedPartitionMsckRepairTest` (11 SQL-level cases) stays green. ### API and Format No API or format change. `partitionFilter` and `partitionType` stay null on purpose: repair diffs raw directory names, and handing it a partition type would reintroduce the cast that rewrites `month=01` to `1` and no longer round-trips to the real directory. A comment now records that reason so the two nulls do not get "helpfully" filled in later. ### Note on scope This fixes the default partition name. A partition value that merely *starts* with `_` or `.` (`dt='_abc'`) is still dropped by SYNC in a value-only layout, because `_` and `.` are not in `CHAR_TO_ESCAPE` and the directory is written bare. Widening the rule is a separate discussion: in a value-only layout a partition directory and a committer staging directory sit at the same level and cannot be told apart by name, so relaxing the rule would register `_temporary`/`__magic` trees as partitions. Happy to open a separate issue for that if you agree it needs a decision. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
