JingsongLi commented on code in PR #11017:
URL: https://github.com/apache/paimon/pull/11017#discussion_r4583976228
##########
paimon-core/src/main/java/org/apache/paimon/operation/RestoreFiles.java:
##########
@@ -43,12 +45,31 @@ public RestoreFiles(
@Nullable IndexFileMeta dynamicBucketIndex,
@Nullable List<IndexFileMeta> deleteVectorsIndex,
@Nullable List<IndexFileMeta> sourceIndexPayloads) {
+ this(
+ snapshot,
+ totalBuckets,
+ dataFiles,
+ dynamicBucketIndex,
+ deleteVectorsIndex,
+ sourceIndexPayloads,
+ Collections.emptyList());
Review Comment:
[P1] Carry guards through Flink coordinated restore
The six-argument compatibility constructor silently supplies empty guards,
but CoordinatedWriteRestore still uses it and neither TableWriteCoordinator nor
ScanCoordinationResponse transports guard references.
sink.writer-coordinator.enabled=true is accepted together with the new feature,
and both FixedBucketSink and CompactorSink select this restore path without a
guard exclusion. I reproduced this using real Parquet files and the actual
coordinator's paged serialized RPC response (only the TaskOperatorEventGateway
transport is mocked): after capturing a delete-at-4 guard and appending a stale
version2, filesystem restore returns one guard and full compaction succeeds;
coordinated restore returns zero guards and the same compaction commit throws
'Delete ordering guard sources changed ... bucket 0'. Reloading through the
same coordinator still omits the active guard, so the configured sink/dedicated
compactor cannot progress normally once guards exist. Carry the exact
snapshot's gu
ards through the coordinator response and restore path, or reject this option
combination until it is supported; retain the publication fence.
##########
paimon-python/pypaimon/table/delete_ordering_guards.py:
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Reject guarded histories in unsupported clients."""
+
+_UNSET = object()
+_ENABLED = 'delete-ordering-guards.enabled'
+_PREFIX = 'delete-ordering-guards.'
+_ERROR = (
+ 'PyPaimon does not support delete ordering guards. '
+ 'Use a guard-aware Java client to read or write this table; '
+ 'disabling capture does not remove retained guard history.'
+)
+
+
+def validate_delete_ordering_guards(table, snapshot=_UNSET):
+ if not table.is_primary_key_table:
+ return
+ options = table.options.options.to_map()
+ validate_delete_ordering_guard_history(options, None)
+ if snapshot is _UNSET:
+ snapshot = table.snapshot_manager().get_latest_snapshot()
Review Comment:
[P2] Preserve Native JDBC planning without a Python catalog connection
This unconditionally loads the latest snapshot before selecting the Native
planner/reader, including ordinary PK tables with guard capture disabled and no
guard history. A JDBC-backed SnapshotManager first calls SnapshotLoader ->
JdbcCatalogLoader.load, opening a fresh Python JDBC connection, before the
unsupported catalog snapshot loader can fall back to files. Previously
resolved-schema Native planning did not need that Python catalog
connection/driver. The unchanged real Native native_plan_resolved_schema_test
now fails four pk-jdbc cases with 'Failed to load snapshot: JDBC connection
during planning'; all 82 cases pass on clean ba9b1bc with the same Rust0.5.0
runtime, and all four Native CI lanes report the same failures on this head.
This introduces a new Python-driver/catalog-availability dependency for normal
Native PK reads. Validate guard history through the resolved/Native snapshot
path, or avoid constructing the unsupported JDBC snapshot loader, while
preserving rejecti
on of genuinely guarded histories.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]