XiaoHongbo-Hope opened a new pull request, #9250: URL: https://github.com/apache/paimon/pull/9250
Ray workers may deserialize pickled split objects concurrently when many scan splits are scheduled. Importing a low-level pypaimon submodule currently executes the package root first, whose eager public exports load a broad catalog, filesystem, schema, and read dependency graph. Concurrent imports can then observe partially initialized modules or hit a Python module-lock deadlock. This change lazily resolves the existing root-level public exports on Python 3.7 and newer, while retaining the eager compatibility path for Python 3.6. Public imports such as from pypaimon import CatalogFactory remain unchanged. A fresh-process regression test concurrently imports and deserializes a real QueryAuthSplit containing DataFileMeta instances across 24 threads. Validation: - concurrent import regression repeated 30 times - query-auth, catalog environment, and concurrent import tests: 61 passed - flake8 and git diff --check passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
