JingsongLi commented on code in PR #11097: URL: https://github.com/apache/paimon/pull/11097#discussion_r4536628057
########## paimon-python/pypaimon/multimodal/lerobot/ray_append.py: ########## @@ -0,0 +1,69 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Serialize LeRobot group publication across jobs on one Ray cluster.""" + +import hashlib + +from pypaimon.multimodal.lerobot.metadata import _companion_identifier + + +_NAMESPACE = "pypaimon_lerobot" + + +def _group_identity(connection, table_name): + frames_name = _companion_identifier( + connection._identifier(table_name), "_frames") + frames = connection.get_table(frames_name).raw_table + return frames.catalog_environment.uuid or frames.table_path Review Comment: [P1] Use a canonical identity for filesystem table groups For filesystem catalogs, `catalog_environment.uuid` is `None` and `table_path` retains the warehouse's original spelling. Two jobs using `/shared/warehouse` and `file:///shared/warehouse` therefore generate different actor names for the same physical table group, even though both use `coordinated=True` on the same Ray cluster. I verified this with a real LeRobot group: both connections read and write the same tables, but `_group_identity` differs by the `file://` prefix. The two actors can then run append/recovery concurrently, reintroducing the race this PR is intended to prevent. Please use a persisted table/group ID or canonicalize the storage identity before deriving the actor name. ########## paimon-python/pypaimon/multimodal/lerobot/aggregate.py: ########## @@ -84,6 +85,8 @@ _DATASETS_SCHEMA = pa.schema([ pa.field("index", pa.int64(), nullable=False), pa.field("name", pa.string()), + pa.field("request_id", pa.string()), Review Comment: [P2] Preserve ordinary strict-schema appends to existing groups These columns are now included in every incoming Dataset schema, including direct appends with `request_id=None`. A table group created before this change lacks them, so appending an otherwise compatible source with `schema_evolution=False` now raises `LeRobot source 1 adds field request_id with schema_evolution disabled`, even when coordination is not enabled and no business columns have changed. I reproduced the full import/append path with the previous Dataset schema; the same append succeeds with the baseline schema. Please avoid requiring request metadata for ordinary legacy appends, or handle internal metadata migration separately from feature-schema evolution. ########## paimon-python/pypaimon/multimodal/lerobot/ray_append.py: ########## @@ -0,0 +1,69 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Serialize LeRobot group publication across jobs on one Ray cluster.""" + +import hashlib + +from pypaimon.multimodal.lerobot.metadata import _companion_identifier + + +_NAMESPACE = "pypaimon_lerobot" + + +def _group_identity(connection, table_name): + frames_name = _companion_identifier( + connection._identifier(table_name), "_frames") + frames = connection.get_table(frames_name).raw_table + return frames.catalog_environment.uuid or frames.table_path + + +class _AppendCoordinator: + + def __init__(self, identity): + self._identity = identity + + def append(self, identity, options, database, table_name, source, kwargs): + if identity != self._identity: + raise ValueError("LeRobot append coordinator targets a different table group.") + from pypaimon.multimodal import connect + + connection = connect(database=database, options=options) + if _group_identity(connection, table_name) != identity: + raise ValueError("LeRobot table group changed since append submission.") + # A previous actor may have died after committing Frames or Episodes. + connection.recover_lerobot_append(table_name) + connection.append_to_lerobot(table_name, source, **kwargs) + + +def append_coordinated(connection, table_name, source, *, request_id, **kwargs): + """Route one idempotent append through the cluster-wide group actor.""" + from pypaimon.multimodal.lerobot.ray_loader import _require_ray + + if not isinstance(request_id, str) or not request_id.strip(): + raise ValueError("coordinated LeRobot append requires a non-empty request_id.") + if connection._catalog_options is None: + raise ValueError("coordinated LeRobot append requires connect(options=...).") + ray = _require_ray() + identity = _group_identity(connection, table_name) + name = "lerobot_append_" + hashlib.sha256(identity.encode("utf-8")).hexdigest() + actor = ray.remote(_AppendCoordinator).options( + name=name, namespace=_NAMESPACE, lifetime="detached", + get_if_exists=True, max_concurrency=1, num_cpus=1, Review Comment: [P1] Avoid reserving a CPU for the coordinator's lifetime This detached actor keeps its explicit `num_cpus=1` allocation even while idle or waiting for preparation tasks. Those tasks also require one CPU in `_prepare_sources_ray`, so a coordinated append with `parallelism > 1` cannot make progress on a one-CPU cluster. On a larger cluster, coordinators for enough groups can consume every CPU and block further preparation or actor creation. I reproduced the scheduling deadlock with these actor options on Ray 2.59.1; changing only the coordinator to `num_cpus=0` allowed the child task to finish. Please avoid retaining a CPU for the coordinator's lifetime, or otherwise manage its resources so preparation can always run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
