GitHub user zyratlo edited a discussion: Merge the Python Notebook Migration
Tool in a Single PR
## Summary
A new feature currently in the review phase before being merged is the Python
Notebook Migration Tool. The related pages are Issue #4301 and PR #4486 . As
of the time of this writing, the PR has 37 files changed and 3,342 additions.
We have decided to currently merge this large addition to the code base in a
single PR for multiple reasons:
- The code is not able to be sliced vertically to merge in multiple PR's
- While portions of the code can be merged across multiple PR's, since we
want the tool to still be functional in each PR we cannot divide it while
maintaining functionality
- The code can be merged safely in one PR since it does not interfere or modify
with any other parts of the code base.
- Existing files which are modified are only changed to add functionality
for this tool, and ensures any other pre-existing code is unchanged
- A full breakdown of the new code can be found below
## Architecture
As shown in the architecture diagrams below, a large portion of the new code is
contained within new files. Portions of the front-end and existing services
(such as the LiteLLM proxy) are reused while maintaining their original
purposes are unchanged.
<img width="1333" height="942" alt="Image"
src="https://github.com/user-attachments/assets/ce31d2ed-2aad-402c-9ec7-02d58ec53980";
/>
<img width="1332" height="935" alt="Image"
src="https://github.com/user-attachments/assets/b34491a3-9033-4e93-a303-07806079ab07";
/>
<img width="1343" height="938" alt="Image"
src="https://github.com/user-attachments/assets/a3e6d77a-e369-40dd-85f5-47be4eb2d4f9";
/>
## Breakdown of Code Changes
More detailed breakdown can be found
[here](https://docs.google.com/document/d/1HnE2g2fDXnCw6Qm072_cNZyv4zZ4XRL_AIAEgJ_pYXc/edit?usp=sharing)
### SQL
- `sql/updates/23.sql`
`sql/texera_ddl.sql`
- Add notebook and workflow_notebook_mapping tables to database
### Modified Existing Files
#### Config files
- `build.sbt`
- Add NotebookMigrationService
- `common/config/src/main/resources/storage.conf`
`common/config/src/main/scala/org/apache/texera/amber/config/StorageConfig.scala`
- Added path for Jupyter server
- `frontend/proxy.config.json`
- Add notebook-migration proxy
- `frontend/src/app/app.module.ts`
- Add Jupyter components/services
#### Front-end
- `frontend/src/app/workspace/component/menu/menu.component.ts`
- Added new button for upload popup and calling necessary services for the
tool
-
`frontend/src/app/workspace/component/workflow-editor/mini-map/workspace.component.ts`
- Added spinner and stopwatch for LLM waiting phase
### New Files
#### Front-end
-
`frontend/src/app/workspace/component/jupyter-notebook-panel/jupyter-notebook-panel.component.ts`
- Logic to handle the Jupyter iframe (opening, closing, find the iframe
URL, etc.)
- `frontend/src/app/workspace/service/jupyter-panel/jupyter-panel.service.ts`
- Handles communication between Jupyter iframe and the workflow
- `frontend/src/app/workspace/service/notebook-migration/migration-llm.ts`
- Handles all communication with LiteLLM proxy
-
`frontend/src/app/workspace/service/notebook-migration/notebook-migration.service.ts`
- Handles communication with back-end and mapping logic
#### Back-end
- `notebook-migration-service/`
- All back-end code is in this directory and no code outside of it has been
modified
GitHub link: https://github.com/apache/texera/discussions/4590
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
