JingsongLi commented on code in PR #10723:
URL: https://github.com/apache/paimon/pull/10723#discussion_r4491812735


##########
paimon-python/pypaimon/filesystem/vortex_utils.py:
##########
@@ -67,3 +69,97 @@ def to_vortex_specified(file_io: FileIO, file_path: str) -> 
Tuple[str, Optional[
         file_path_for_vortex = file_path_for_vortex.replace('oss://', 's3://')
 
     return file_path_for_vortex, store_kwargs
+
+
+# Paimon never rewrites a data file in place, and every data file has a unique 
name. So a footer,
+# read once, stays valid for the path, and a later open of the same file needs 
no footer IO.
+_FOOTER_CACHE_SIZE = 1024
+_footer_cache: "OrderedDict[str, Any]" = OrderedDict()
+_store_cache: Dict[Tuple[str, Tuple[Tuple[str, str], ...]], Any] = {}
+_cache_lock = threading.Lock()

Review Comment:
   [P2] Reinitialize the cache lock in forked DataLoader workers
   
   If another thread owns this new module lock when a DataLoader forks its 
workers (the usual Linux/Python 3.11 setup), each child inherits a locked mutex 
whose owning thread no longer exists. Its first Python-backend Vortex read 
blocks in `_cached_footer`, even after the parent releases the lock. 
`_store_for` also holds this lock while creating a native store client, 
extending the fork window.
   
   I reproduced this through the public `table_read.to_torch(..., 
streaming=True)` and `DataLoader` APIs with a real Vortex 0.91 table: the 
unmodified worker timed out; with the same table, SDK and fork schedule, 
replacing only `_cache_lock` in worker initialization returned `[1, 2, 3]`. The 
forking parent had never imported Vortex, excluding inherited native-runtime 
state as the cause. Please reinitialize this lock after fork (for example, an 
after-in-child hook like the existing FileIO fork handling) and cover this 
interleaving with a worker-read regression.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to