zyratlo opened a new pull request, #8767: URL: https://github.com/apache/texera/pull/8767
### What changes were proposed in this PR? The migration tool accepted a Jupyter notebook and a single Python file. It now accepts a folder of Python files selected from disk, and produces one workflow from all of them, with the same stored source and operator mapping that drive highlighting in the Jupyter panel. A folder is harder than a script for the same reason a script was harder than a notebook: the mapping keys on cells, and there are none. This PR does not introduce a second mechanism for that. The folder's Python files are concatenated into one document, each behind a banner line naming the file it came from, with line numbers running continuously across the whole thing. The model then reports line ranges exactly as it does for a single script, and #8007's segmenter derives the cells. The mapping stays keyed on cell uuids, so storage, the Jupyter panel and the highlight index never learn that folders exist. No backend, database, Helm, or dependency changes. The alternative was to number each file separately and key the mapping on file plus line range. That changes the stored mapping shape, which pulls in the backend, the database and the panel, to buy provenance the banner line already provides. Main pieces: * `folder-assembly.ts`, a new pure module: which files are in scope, the order they are joined in, the assembled document, the layout, the caps, and entry point resolution. It reuses the segmenter's own line splitting, so assembling, numbering the prompt and slicing the cells all agree on what a line is. * `segmentScript` gains one optional argument, a list of caller-required cut points. Each file's banner joins the boundary set, so no cell holds lines from two files. Every invariant the module already documented holds unchanged. * An indented layout of the folder, sent as prompt text ahead of the numbered document. It is deliberately not part of that document: numbering it would shift every line the model reports, and the segmenter would emit it as a cell of directory listing. It also names the folder's non-Python files without reading them, so the model knows a dataset or a requirements list exists. * Folder variants of the conversion prompt, the mapping prompt, and the worked example. `migration-prompts.ts` is **232 insertions and 0 deletions**: the notebook and script prompts are untouched, which a check in review verified byte for byte. The derived notebook holds the entry point alone, not every file. A folder's other files are function definitions, and a notebook of all of them reads as a wall of code rather than the story of what the project does. The model names the entry point in the conversion reply, so this costs no extra round trip, and the mapping it reports is expressed in that file's lines. Clicking an operator highlights the call that runs it. When the entry point cannot be resolved the notebook falls back to the whole folder, which is worse to read but never empty. Two caps refuse an oversized folder before any request is sent: 100 files, and 60,000 characters of code. The binding constraint is the reply rather than the prompt, because the conversion asks the model to return every line of code as JSON-escaped strings and no output budget is set. A cap the model cannot meet would defeat the point of refusing up front, since the user would wait out a full conversion and get a JSON parse error from a truncated reply. A byte check runs before anything is read, so one enormous generated file cannot be loaded into memory first. Dropping a folder onto the upload button is refused with a message. ng-zorro binds a drop handler on the button whatever its type, and in directory mode it hands over one file at a time with no path attached, so accepting a drop would stage a single arbitrary file and convert it as if it were the whole project. Three things ride along: * The three import modal diagrams are normalized to one size. They were the same layout exported at three resolutions with uneven whitespace, so the tabs rendered at slightly different scales. Trimming each to its content and scaling to a common canvas touches two images from earlier PRs. * `LICENSE-binary` credits `snippet_folder` for the new diagram, and adds `python_file_diagram.png` to the Material Symbols paths. That second entry fills an attribution gap from #8007, where the same icons were reused without extending the path list. * A final commit consolidates the folder caps and the `webkitRelativePath` parsing, which had spread across two files while the feature was built. #### Demo video https://drive.google.com/file/d/1KtI2FVWgOwoGCFZfnIWXLGurBcXQmmkz/view?usp=sharing ### Any related issues, documentation, discussions? Closes #8008 Parent-issue #4301 The reasoning for keeping the mapping and Jupyter panel on an input without cells was settled for scripts in https://github.com/apache/texera/discussions/8489 and applies to folders unchanged. The output budget argument behind the character cap applies just as much to a single `.py` and to a notebook, neither of which has a size cap today, and joining a project into one script is an easy way around the folder cap. That is worth its own issue rather than widening this one. ### How was this PR tested? Unit tests: * `folder-assembly.spec.ts`, new, 59 tests. File selection, ordering, assembly, the layout, the caps, entry point resolution and scoping. Two cases are worth naming. Directories sort by segment, so a sibling whose name sorts between a directory and its subdirectory (`src`, `src-old`, `src/utils`) cannot be rendered as the wrong parent. And an ambiguous entry point name resolves to null rather than to whichever file sorts first, because showing the wrong file is worse than falling back to the whole folder. * `script-segmentation.spec.ts`, 32 to 36. Forced cut points, including that out of range and non-integer boundaries are dropped rather than clamped. * `migration-llm.spec.ts`, 42 to 57. The layout is sent ahead of the code and never numbered, cuts land on every file banner, a range straddling a seam still maps to the cells on both sides, and the entry point scopes the notebook with a fallback when it cannot be resolved. * `notebook-migration.service.spec.ts`, 48 to 65. Reading a selection, the layout it produces, and each cap, including that the file count and byte checks run before any file is read. * `notebook-import-modal.component.spec.ts`, 27 to 39. The third tab, the directory picker, and the drop path modelled the way a real drop arrives: one file per call, each in its own one-element list, with no path. * `user-workflow.component.spec.ts`, 81 to 87. Dispatch on selection shape, and that a folder names its workflow after itself. Manually tested end to end against a live model with a small multi-file project (a package with an entry point, two modules and a helpers-only module, plus non-Python files and a `__pycache__`). ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
