JingsongLi commented on code in PR #11097:
URL: https://github.com/apache/paimon/pull/11097#discussion_r4536628057


##########
paimon-python/pypaimon/multimodal/lerobot/ray_append.py:
##########
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Serialize LeRobot group publication across jobs on one Ray cluster."""
+
+import hashlib
+
+from pypaimon.multimodal.lerobot.metadata import _companion_identifier
+
+
+_NAMESPACE = "pypaimon_lerobot"
+
+
+def _group_identity(connection, table_name):
+    frames_name = _companion_identifier(
+        connection._identifier(table_name), "_frames")
+    frames = connection.get_table(frames_name).raw_table
+    return frames.catalog_environment.uuid or frames.table_path

Review Comment:
   [P1] Use a canonical identity for filesystem table groups
   
   For filesystem catalogs, `catalog_environment.uuid` is `None` and 
`table_path` retains the warehouse's original spelling. Two jobs using 
`/shared/warehouse` and `file:///shared/warehouse` therefore generate different 
actor names for the same physical table group, even though both use 
`coordinated=True` on the same Ray cluster. I verified this with a real LeRobot 
group: both connections read and write the same tables, but `_group_identity` 
differs by the `file://` prefix. The two actors can then run append/recovery 
concurrently, reintroducing the race this PR is intended to prevent. Please use 
a persisted table/group ID or canonicalize the storage identity before deriving 
the actor name.



##########
paimon-python/pypaimon/multimodal/lerobot/aggregate.py:
##########
@@ -84,6 +85,8 @@
 _DATASETS_SCHEMA = pa.schema([
     pa.field("index", pa.int64(), nullable=False),
     pa.field("name", pa.string()),
+    pa.field("request_id", pa.string()),

Review Comment:
   [P2] Preserve ordinary strict-schema appends to existing groups
   
   These columns are now included in every incoming Dataset schema, including 
direct appends with `request_id=None`. A table group created before this change 
lacks them, so appending an otherwise compatible source with 
`schema_evolution=False` now raises `LeRobot source 1 adds field request_id 
with schema_evolution disabled`, even when coordination is not enabled and no 
business columns have changed. I reproduced the full import/append path with 
the previous Dataset schema; the same append succeeds with the baseline schema. 
Please avoid requiring request metadata for ordinary legacy appends, or handle 
internal metadata migration separately from feature-schema evolution.



##########
paimon-python/pypaimon/multimodal/lerobot/ray_append.py:
##########
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Serialize LeRobot group publication across jobs on one Ray cluster."""
+
+import hashlib
+
+from pypaimon.multimodal.lerobot.metadata import _companion_identifier
+
+
+_NAMESPACE = "pypaimon_lerobot"
+
+
+def _group_identity(connection, table_name):
+    frames_name = _companion_identifier(
+        connection._identifier(table_name), "_frames")
+    frames = connection.get_table(frames_name).raw_table
+    return frames.catalog_environment.uuid or frames.table_path
+
+
+class _AppendCoordinator:
+
+    def __init__(self, identity):
+        self._identity = identity
+
+    def append(self, identity, options, database, table_name, source, kwargs):
+        if identity != self._identity:
+            raise ValueError("LeRobot append coordinator targets a different 
table group.")
+        from pypaimon.multimodal import connect
+
+        connection = connect(database=database, options=options)
+        if _group_identity(connection, table_name) != identity:
+            raise ValueError("LeRobot table group changed since append 
submission.")
+        # A previous actor may have died after committing Frames or Episodes.
+        connection.recover_lerobot_append(table_name)
+        connection.append_to_lerobot(table_name, source, **kwargs)
+
+
+def append_coordinated(connection, table_name, source, *, request_id, 
**kwargs):
+    """Route one idempotent append through the cluster-wide group actor."""
+    from pypaimon.multimodal.lerobot.ray_loader import _require_ray
+
+    if not isinstance(request_id, str) or not request_id.strip():
+        raise ValueError("coordinated LeRobot append requires a non-empty 
request_id.")
+    if connection._catalog_options is None:
+        raise ValueError("coordinated LeRobot append requires 
connect(options=...).")
+    ray = _require_ray()
+    identity = _group_identity(connection, table_name)
+    name = "lerobot_append_" + 
hashlib.sha256(identity.encode("utf-8")).hexdigest()
+    actor = ray.remote(_AppendCoordinator).options(
+        name=name, namespace=_NAMESPACE, lifetime="detached",
+        get_if_exists=True, max_concurrency=1, num_cpus=1,

Review Comment:
   [P1] Avoid reserving a CPU for the coordinator's lifetime
   
   This detached actor keeps its explicit `num_cpus=1` allocation even while 
idle or waiting for preparation tasks. Those tasks also require one CPU in 
`_prepare_sources_ray`, so a coordinated append with `parallelism > 1` cannot 
make progress on a one-CPU cluster. On a larger cluster, coordinators for 
enough groups can consume every CPU and block further preparation or actor 
creation. I reproduced the scheduling deadlock with these actor options on Ray 
2.59.1; changing only the coordinator to `num_cpus=0` allowed the child task to 
finish. Please avoid retaining a CPU for the coordinator's lifetime, or 
otherwise manage its resources so preparation can always run.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to