GitHub user aicam edited a discussion: Use Mounting Feature for Native Operators
## 📌 Context & Current State We recently implemented the **LakeFS repository mounting** feature for Python UDFs. Currently, mounting a LakeFS repository into a workflow is strictly accessible via Python execution inside UDFs. While this works well for custom Python scripts, the mounting capability can provide immense value if exposed directly to **native operators** within Computing Units (CUs) during workflow execution. --- ## 🎯 Problem Statement & Opportunity Previously, native operators faced several limitations when dealing with LakeFS or S3 datasets: * **Multi-file traversal:** A single operator could not easily traverse directory trees or iterate over multiple files belonging to the same dataset. * **Temp file overhead:** Data often had to be explicitly downloaded or copied into temporary local files before processing. By mounting a LakeFS repository directly inside a Computing Unit (CU), the entire dataset version becomes accessible as a native filesystem path throughout the workflow run. --- ## 💡 Proposed Use Cases for Native Operators We want to explore different ways native operators can leverage mounted LakeFS repositories. Here are a few initial ideas: ### 1. Pattern-Based Scan Operators Scan operators could accept a dataset version path (or subfolder) along with a glob pattern (e.g., `*.csv`, `data_*.parquet`). Depending on configuration, the operator could: * Concatenate matching files into a single unified stream. * Emit one row/record per discovered file (useful for file-level metadata processing). ### 2. "List Files" Source Operator A dedicated source operator that scans a mounted directory and outputs file paths/metadata downstream. Downstream operators can then process individual files in parallel or sequentially. ### 3. In-Place File Processing Operators can read mounted files directly via standard filesystem interfaces (`/mnt/lakefs/...`) rather than executing expensive pre-copying operations to local `/tmp` directories. ### 4. Dynamic Multi-Port Output Allow a scan operator to dynamically generate output ports based on the number of matching files discovered during workflow setup or runtime execution. --- ## ⚡ Benefits: Mounting vs. Direct S3 API Calls Why leverage filesystem mounting instead of standard S3 API client calls? | Feature | Direct S3 API Calls | Mounted LakeFS Repository | | :--- | :--- | :--- | | **Interface** | Requires S3 SDK integrations per operator | Standard OS & POSIX filesystem calls (`open()`, `read()`, `ls`) | | **Data Fetching** | Eager download or explicit range queries | **Lazy loading:** Data blocks are fetched on-demand | | **Resource Efficiency** | Downloads full files even if partially read | Efficient when querying/inspecting large multi-file datasets | --- ## 💬 Questions for the Community We would love to get your feedback on this proposal: 1. **Which native operator patterns would benefit your current workflows the most?** 2. **Are there performance or schema considerations** we should keep in mind for dynamic multi-port scan operators? 3. **Are there other use cases** for mounted repositories in native workflows that we haven't considered yet? GitHub link: https://github.com/apache/texera/discussions/9097 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
