GitHub user aicam created a discussion: Reading multiple dataset files in native operators: UX ideas
Following @chenlica's suggestion in #9048, this thread is only about the user experience of reading datasets in native operators. Security and the Python library are left for later. ### Where we are Each scan operator (CSV, JSON, Arrow, Text, File Scan) takes one file from one dataset version. Reading many files means one scan operator per file, or a Python UDF. Since #6896 and #6897, the engine can mount a whole dataset or model version into the computing unit, but only Python UDFs ask for it. The hook is generic, so any operator could declare which versions it needs. "File Scan From Input" already reads file paths that come from upstream tuples, which is a building block we can reuse. @zyratlo raised a point in #9048 that I think should be a ground rule: a workflow should not behave differently depending on how Texera is deployed. Mounting is off by default, so it should only make reads faster when it is available, never decide whether a workflow works. ### Ideas **1. Folder or pattern input on scan operators.** The file picker accepts a version plus a folder or a glob such as `train/*.csv`. The operator reads every match as one table, with an optional "file name" column. The matches are resolved when the workflow is compiled and are fixed to the selected version, so the result is reproducible. The property panel shows how many files match and lists a few. Zero matches, or files whose schema does not match the first one, give a compile error. **2. A "Dataset Files" source operator.** It lists the files of a version as tuples (path, name, size, extension). Users filter, sort or sample them with existing operators and pass the result to "File Scan From Input" or a UDF. This is more flexible than a pattern and adds no new syntax. **3. Folder-level operators.** Some operators need a directory, not rows: an image folder for an ML operator, a model with several weight files, or a shapefile with its sidecar files. These operators take a version or folder as a parameter and receive a local directory at run time. **4. One read path, two backends.** The operator only declares "I need these files from version X". The engine mounts them when mounting is enabled and otherwise downloads or streams them, so the operator code is the same either way. Listing for ideas 1 and 2 uses the dataset API and never needs a mount. ### Questions 1. Which of these would users reach for first: a pattern in the scan operators (1), a listing operator (2), or both? 2. For patterns, should we allow only simple globs (`*`, `**`, `?`) and validate them, or accept any pattern the user writes? (from @zyratlo) 3. When files in a folder have different schemas, should we fail, take the union of columns, or only read the files that match the first one? 4. Which workloads need a whole folder (idea 3)? Image datasets and model weights come to mind. What else? 5. Is idea 4 the right contract, with mounting as an optimization only? Are there cases where an operator should require a mount? 6. Should one operator read from several dataset versions or datasets at once, or is one version per operator enough? GitHub link: https://github.com/apache/texera/discussions/9097 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
