JingsongLi commented on code in PR #10723:
URL: https://github.com/apache/paimon/pull/10723#discussion_r4491812735
##########
paimon-python/pypaimon/filesystem/vortex_utils.py:
##########
@@ -67,3 +69,97 @@ def to_vortex_specified(file_io: FileIO, file_path: str) ->
Tuple[str, Optional[
file_path_for_vortex = file_path_for_vortex.replace('oss://', 's3://')
return file_path_for_vortex, store_kwargs
+
+
+# Paimon never rewrites a data file in place, and every data file has a unique
name. So a footer,
+# read once, stays valid for the path, and a later open of the same file needs
no footer IO.
+_FOOTER_CACHE_SIZE = 1024
+_footer_cache: "OrderedDict[str, Any]" = OrderedDict()
+_store_cache: Dict[Tuple[str, Tuple[Tuple[str, str], ...]], Any] = {}
+_cache_lock = threading.Lock()
Review Comment:
[P2] Reinitialize the cache lock in forked DataLoader workers
If another thread owns this new module lock when a DataLoader forks its
workers (the usual Linux/Python 3.11 setup), each child inherits a locked mutex
whose owning thread no longer exists. Its first Python-backend Vortex read
blocks in `_cached_footer`, even after the parent releases the lock.
`_store_for` also holds this lock while creating a native store client,
extending the fork window.
I reproduced this through the public `table_read.to_torch(...,
streaming=True)` and `DataLoader` APIs with a real Vortex 0.91 table: the
unmodified worker timed out; with the same table, SDK and fork schedule,
replacing only `_cache_lock` in worker initialization returned `[1, 2, 3]`. The
forking parent had never imported Vortex, excluding inherited native-runtime
state as the cause. Please reinitialize this lock after fork (for example, an
after-in-child hook like the existing FileIO fork handling) and cover this
interleaving with a worker-read regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]