Copilot commented on code in PR #8544:
URL: https://github.com/apache/texera/pull/8544#discussion_r4008361641
##########
frontend/src/app/workspace/component/notebook-import-modal/notebook-import-modal.component.html:
##########
@@ -17,69 +17,130 @@
under the License.
-->
-<div class="import-modal-diagram">
- <img
- ngSrc="assets/notebook_migration_tool/tool_popup_diagram.png"
- alt="Notebook to Workflow"
- width="1132"
- height="290" />
-</div>
-
<div class="import-modal-content">
<form
class="import-modal-form"
[formGroup]="importForm"
[attr.inert]="isSubmitting ? '' : null"
nz-form>
- <nz-form-item>
- <p class="import-modal-text">
- This tool converts a Python Jupyter Notebook into a Texera workflow
using LLM capabilities. After you submit a
- notebook, the LLM service generates a corresponding Texera workflow.
The conversion time depends on the
- notebook's complexity and can take 1-5 minutes. Once generation
finishes, you are taken to the new workflow,
- which opens with:
- </p>
- <ol class="import-modal-list">
- <li>
- The generated workflow ready to use (Note: you will still need to
upload the dataset and connect it to the
- workflow).
- </li>
- <li>A floating Jupyter window containing the uploaded notebook for
reference.</li>
- </ol>
- <p class="import-modal-text">
- Generation runs here after you submit. Please keep this window open
while you wait.
- </p>
- </nz-form-item>
+ <!-- One upload row shared by both tabs. They differ only in the accepted
extension and
+ their labels, so the markup lives here once and each tab passes its
own context. -->
+ <ng-template
+ #uploadRow
+ let-accept="accept"
+ let-label="label"
+ let-action="action">
+ <nz-form-item>
+ <nz-form-label [nzNoColon]="true">
+ <span class="import-modal-label">{{ label }}</span>
+ </nz-form-label>
+ <nz-form-control>
+ <div class="import-modal-upload-row">
+ <nz-upload
+ [nzAccept]="accept"
+ [nzBeforeUpload]="beforeUpload"
+ [nzShowUploadList]="false">
+ <button
+ nz-button
+ type="button"
+ [title]="action"
+ [attr.aria-label]="action">
+ <i
+ nz-icon
+ nzType="upload"></i>
+ </button>
+ </nz-upload>
- <nz-form-item>
- <nz-form-label [nzNoColon]="true">
- <span class="import-modal-label"> Upload Python Jupyter Notebook
</span>
- </nz-form-label>
- <nz-form-control>
- <div class="import-modal-upload-row">
- <nz-upload
- nzAccept=".ipynb"
- [nzBeforeUpload]="beforeUpload"
- [nzShowUploadList]="false">
- <button
- nz-button
- type="button"
- title="Upload notebook"
- aria-label="Upload notebook">
- <i
- nz-icon
- nzType="upload"></i>
- </button>
- </nz-upload>
+ <span
+ *ngIf="importForm.get('file')?.value?.name as fileName"
+ class="import-modal-selected-file"
+ [title]="fileName">
+ Selected file: {{ fileName }}
+ </span>
+ </div>
+ </nz-form-control>
+ </nz-form-item>
+ </ng-template>
- <span
- *ngIf="importForm.get('file')?.value?.name as fileName"
- class="import-modal-selected-file"
- [title]="fileName">
- Selected file: {{ fileName }}
- </span>
- </div>
- </nz-form-control>
- </nz-form-item>
+ <nz-tabs
+ [nzSelectedIndex]="selectedTabIndex"
+ [nzAnimated]="tabAnimation"
+ (nzSelectedIndexChange)="onTabChange($event)">
Review Comment:
This adds a visible tabbed modal, but the PR description does not include
the repository-required before/after screenshots or GIF for frontend UI
changes. Please add visual evidence (plus the stated manual flow) to the PR's
testing section so reviewers can assess the layout in both tabs.
##########
frontend/src/app/workspace/service/notebook-migration/script-segmentation.ts:
##########
@@ -0,0 +1,191 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { v4 as uuidv4 } from "uuid";
+
+/**
+ * Turns a Python script into notebook-style cells using the line ranges the
LLM reported
+ * for each UDF.
+ *
+ * A notebook arrives already split into cells, and those cells are the join
key for the
+ * cell<->operator mapping that drives highlighting. A script has no such
boundaries, so the
+ * model is asked which lines it turned into which UDF, and the cells are
derived from that
+ * answer here.
+ *
+ * The model's answer is untrusted: ranges can arrive reversed, overlapping,
out of order,
+ * past the end of the file, or missing entirely. Every case is reconciled
rather than
+ * rejected, because the ranges arrive alongside a workflow that already cost
a full
+ * conversion, and a degraded mapping is worth more than a discarded result.
+ *
+ * Guarantees, given a non-empty script:
+ * - Every line with content lands in exactly one cell, whether or not a UDF
claimed it.
+ * - Cells are disjoint and ordered by position in the file.
+ * - No cell straddles a reported boundary, so a cell claimed by a UDF is
claimed whole.
+ * - Lines two UDFs both claim become one shared cell that maps to both,
which is what
+ * `cell_to_operator` already expresses for notebooks.
+ */
+
+// A 1-indexed, inclusive span of source lines.
+interface LineRange {
+ start: number;
+ end: number;
+}
+
+export interface DerivedCell {
+ uuid: string;
+ source: string;
+ // 1-indexed and inclusive, retained so callers can report or debug the
split.
+ startLine: number;
+ endLine: number;
+}
+
+export interface ScriptSegmentation {
+ cells: DerivedCell[];
+ // UDF id -> the uuids of the cells it covers. A UDF whose ranges were all
unusable is absent.
+ udfToCellUuids: Record<string, string[]>;
+}
+
+/**
+ * Splits into lines, tolerating CRLF and a trailing newline (which is a
terminator, not a line).
+ *
+ * Exported because the caller numbers these same lines when it builds the
prompt. The numbers the
+ * model reports back are only meaningful if both sides agree on what counts
as a line, so they must
+ * share one definition rather than each keep their own.
+ */
+export function splitScriptLines(source: string): string[] {
+ const lines = source.replace(/\r\n?/g, "\n").split("\n");
+ if (lines.length > 0 && lines[lines.length - 1] === "") {
+ lines.pop();
+ }
+ return lines;
+}
+
+// Accepts a number or a numeric string, since models drift between the two.
+function parseBound(value: unknown): number | null {
+ if (typeof value === "number") {
+ return Number.isFinite(value) ? Math.trunc(value) : null;
+ }
+ if (typeof value === "string" && value.trim() !== "") {
+ const parsed = Number(value);
+ return Number.isFinite(parsed) ? Math.trunc(parsed) : null;
+ }
+ return null;
+}
+
+// Reads one range in either the [start, end] or { start, end } form.
+function toLineRange(raw: unknown): LineRange | null {
+ let rawStart: unknown;
+ let rawEnd: unknown;
+
+ if (Array.isArray(raw)) {
+ if (raw.length !== 2) return null;
+ [rawStart, rawEnd] = raw;
+ } else if (typeof raw === "object" && raw !== null) {
+ ({ start: rawStart, end: rawEnd } = raw as { start?: unknown; end?:
unknown });
+ } else {
+ return null;
+ }
+
+ const start = parseBound(rawStart);
+ const end = parseBound(rawEnd);
+ if (start === null || end === null) return null;
+
+ // A reversed range still names the span the model meant, so read it rather
than drop it.
+ return start <= end ? { start, end } : { start: end, end: start };
+}
+
+// A UDF's ranges may arrive as a list of ranges, a single bare [start, end]
pair, or one
+// { start, end } object. Normalizes all three to a list before parsing.
+function toRangeList(raw: unknown): unknown[] {
+ if (Array.isArray(raw)) {
+ const isBarePair = raw.length === 2 && raw.every(v => typeof v ===
"number" || typeof v === "string");
+ return isBarePair ? [raw] : raw;
+ }
+ return typeof raw === "object" && raw !== null ? [raw] : [];
+}
+
+function clampToSource(range: LineRange, lineCount: number): LineRange | null {
+ const start = Math.max(range.start, 1);
+ const end = Math.min(range.end, lineCount);
+ // Null when the range sat entirely past either end of the file.
+ return start <= end ? { start, end } : null;
+}
+
+/**
+ * @param source the raw script contents
+ * @param rawRanges the model's reply, UDF id -> reported line ranges,
unvalidated
+ * @param newUuid cell id factory, overridden by tests to keep output
deterministic
+ */
+export function segmentScript(
+ source: string,
+ rawRanges: Record<string, unknown> | null | undefined,
+ newUuid: () => string = uuidv4
+): ScriptSegmentation {
+ const lines = splitScriptLines(source);
+ if (lines.length === 0) {
+ return { cells: [], udfToCellUuids: {} };
+ }
+
+ const udfRanges = new Map<string, LineRange[]>();
+ for (const [udfId, raw] of Object.entries(rawRanges ?? {})) {
+ const ranges = toRangeList(raw)
+ .map(toLineRange)
+ .filter((range): range is LineRange => range !== null)
+ .map(range => clampToSource(range, lines.length))
+ .filter((range): range is LineRange => range !== null);
+ if (ranges.length > 0) {
+ udfRanges.set(udfId, ranges);
+ }
+ }
+
+ // Cut the file at every reported boundary. Slicing on the union of
boundaries is what makes
+ // overlaps and gaps fall out on their own: the result is disjoint, covers
every line, and no
+ // segment can span a boundary, so range membership below is an exact
containment test.
+ const boundaries = new Set<number>([1, lines.length + 1]);
+ for (const ranges of udfRanges.values()) {
+ for (const { start, end } of ranges) {
+ boundaries.add(start);
+ boundaries.add(end + 1);
+ }
+ }
+ const cutPoints = [...boundaries].sort((a, b) => a - b);
+
+ const cells: DerivedCell[] = [];
+ for (let i = 0; i < cutPoints.length - 1; i++) {
+ const startLine = cutPoints[i];
+ const endLine = cutPoints[i + 1] - 1;
+ const text = lines.slice(startLine - 1, endLine).join("\n");
+ // A span of only blank lines would render as an empty Jupyter cell, so it
is dropped.
+ // Nothing is lost: every line carrying content still lands in a cell.
+ if (text.trim() === "") continue;
Review Comment:
Dropping boundary-isolated blank spans means the derived notebook no longer
reproduces the uploaded script—for example, the tested blank line 4 is
discarded. This conflicts with the PR's source-fidelity/no-source-loss
guarantee and with the UI saying the notebook contains the user's script.
Preserve blank-only spans (or merge their whitespace without changing mapping
semantics) and cover reassembly with an isolated blank gap.
##########
frontend/src/app/workspace/service/notebook-migration/migration-prompts.ts:
##########
@@ -412,3 +412,173 @@ Here is an example of a mapping generated between the
given example Python code
}
Now create a mapping for the UDFs and the original code. Link the code blocks
marked by 'START <cell-uuid>' and 'END <cell-uuid>' with the UDF UUID's. The
code between them should be equivalent. Multiple cells can be mapped to the
same UDF when that UDF implements the logic of those cells. There could be any
number of cells and UDFs, so only create the correct number in the mapping.
Only give the mapping.
`;
+
+export const EXAMPLE_OF_MULTIPLE_UDF_CONVERSION_SCRIPT = `
+Here is an example of breaking up python code into multiple Texera UDFs.
Format your response structure exactly like the given example. The "code" key
contains a dictionary of the UDF ID's with their respective code. The "edges"
key contains a list of pairs that contains the connections between UDFs. The
"outputs" key contains a dictionary of the UDF ID's with a list of the output
column names of the DataFrame that the UDF yields. The UDFs can branch and
merge, it does not have to be a linear chain depending on your implementation.
+
+The original code is shown with each line prefixed by its line number and a
'|'. Those prefixes are annotations so that line ranges can be referred to
later. They are not part of the code and must never appear in the code you
generate.
+
+Original Code:
+\`\`\`python
+ 1| import pandas as pd
+ 2| from sklearn.model_selection import train_test_split
+ 3| from sklearn.ensemble import RandomForestClassifier
+ 4| from sklearn.svm import SVC
+ 5| from sklearn.tree import DecisionTreeClassifier
+ 6| from sklearn.linear_model import LogisticRegression
+ 7| from sklearn.metrics import accuracy_score
+ 8| from sklearn.preprocessing import StandardScaler
+ 9| import matplotlib.pyplot as plt
+10|
+11| # Load the dataset
+12| file_path = 'diabetes.csv'
+13| data = pd.read_csv(file_path)
+14|
+15| # Remove duplicate rows
+16| data = data.drop_duplicates()
+17|
+18| # Remove rows with null values
+19| data = data.dropna()
+20|
+21| # Print the minimum, maximum, and mean for all fields
+22| print("Minimum values:", data.min())
+23| print("Maximum values:", data.max())
+24| print("Mean values:", data.mean())
+25|
+26| # Create a boxplot for the 'Pregnancies' field
+27| plt.figure(figsize=(8, 6))
+28| plt.boxplot(data['Pregnancies'], vert=False, patch_artist=True)
+29| plt.title('Boxplot of Pregnancies')
+30| plt.xlabel('Number of Pregnancies')
+31| plt.show()
+32|
+33| # Separate features and target variable
+34| X = data.drop('Outcome', axis=1)
+35| y = data['Outcome']
+36|
+37| # Split data into training and testing sets (80% train, 20% test)
+38| X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=42)
+39|
+40| scaler = StandardScaler()
+41| X_train = scaler.fit_transform(X_train)
+42| X_test = scaler.transform(X_test)
+43|
+44| # Train Random Forest model
+45| rf_model = RandomForestClassifier(random_state=42)
+46| rf_model.fit(X_train, y_train)
+47| rf_pred = rf_model.predict(X_test)
+48| rf_accuracy = accuracy_score(y_test, rf_pred)
+49| print(f"Random Forest Accuracy: {rf_accuracy:.2%}")
+50|
+51| # Train SVM model
+52| svm_model = SVC(random_state=42)
+53| svm_model.fit(X_train, y_train)
+54| svm_pred = svm_model.predict(X_test)
+55| svm_accuracy = accuracy_score(y_test, svm_pred)
+56| print(f"SVM Accuracy: {svm_accuracy:.2%}")
+\`\`\`
+
+Texera UDF conversion:
+\`\`\`json
+{
+ "code": {
+ "UDF1": "# UDF1\nfrom pytexera import *\nimport pandas as pd\nfrom
typing import Iterator, Optional\n\nclass
ProcessTableOperator(UDFTableOperator):\n\n @overrides\n def
process_table(self, table: Table, port: int) ->
Iterator[Optional[TableLike]]:\n # Remove duplicate rows\n data =
table.drop_duplicates()\n\n # Remove rows with null values\n data
= data.dropna()\n\n # Calculate statistics\n min_values =
data.min()\n max_values = data.max()\n mean_values =
data.mean()\n\n # Create a DataFrame to yield\n result_table =
pd.DataFrame({\n 'min_values': [min_values],\n
'max_values': [max_values],\n 'mean_values': [mean_values],\n
'data': [data]\n })\n\n yield Table(result_table)",
+ "UDF2": "# UDF2\nfrom pytexera import *\nimport pandas as pd\nimport
plotly.express as px\nimport plotly.io\nfrom typing import Iterator,
Optional\n\nclass ProcessTableOperator(UDFTableOperator):\n def
render_error(self, error_msg):\n return '''<h1>Boxplot is not
available.</h1>\n <p>Reason is: {} </p>\n
'''.format(error_msg)\n\n @overrides\n def process_table(self, table:
Table, port: int) -> Iterator[Optional[TableLike]]:\n data =
table['data'].iloc[0]\n\n if data.empty:\n yield
{'html-content': self.render_error('input table is empty.')}\n
return\n\n # Create a boxplot for the 'Pregnancies' field\n fig =
px.box(data, x='Pregnancies')\n fig.update_layout(margin=dict(l=0, r=0,
t=0, b=0))\n\n # Convert fig to HTML content\n html =
plotly.io.to_html(fig, include_plotlyjs='cdn', auto_play=False)\n yield
{'html-content': html}",
+ "UDF3": "# UDF3\nfrom pytexera import *\nimport pandas as pd\nfrom
sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing
import StandardScaler\nfrom typing import Iterator, Optional\n\nclass
ProcessTableOperator(UDFTableOperator):\n\n @overrides\n def
process_table(self, table: Table, port: int) ->
Iterator[Optional[TableLike]]:\n data = table['data'].iloc[0]\n\n
# Separate features and target variable\n X = data.drop('Outcome',
axis=1)\n y = data['Outcome']\n\n # Split data into training and
testing sets (80% train, 20% test)\n X_train, X_test, y_train, y_test =
train_test_split(X, y, test_size=0.2, random_state=42)\n\n scaler =
StandardScaler()\n X_train = scaler.fit_transform(X_train)\n
X_test = scaler.transform(X_test)\n\n # Create a DataFrame to yield\n
result_table = pd.DataFrame({\n 'X_train': [X_train], 'X_test':
[X_test],\n 'y_train': [y_tra
in], 'y_test': [y_test]\n })\n\n yield Table(result_table)",
+ "UDF4": "# UDF4\nfrom pytexera import *\nimport pandas as pd\nfrom
sklearn.ensemble import RandomForestClassifier\nfrom sklearn.metrics import
accuracy_score\nfrom typing import Iterator, Optional\n\nclass
ProcessTableOperator(UDFTableOperator):\n\n @overrides\n def
process_table(self, table: Table, port: int) ->
Iterator[Optional[TableLike]]:\n X_train = table['X_train'].iloc[0]\n
y_train = table['y_train'].iloc[0]\n X_test =
table['X_test'].iloc[0]\n y_test = table['y_test'].iloc[0]\n\n #
Train Random Forest model\n rf_model =
RandomForestClassifier(random_state=42)\n rf_model.fit(X_train,
y_train)\n rf_pred = rf_model.predict(X_test)\n rf_accuracy =
accuracy_score(y_test, rf_pred)\n\n # Create a DataFrame to yield\n
result_table = pd.DataFrame({\n 'rf_model': [rf_model],\n
'rf_accuracy': [rf_accuracy],\n 'X_test': [X_test],\n
'y_test': [y_test]\n
})\n\n yield Table(result_table)",
+ "UDF5": "# UDF5\nfrom pytexera import *\nimport pandas as pd\nfrom
sklearn.svm import SVC\nfrom sklearn.metrics import accuracy_score\nfrom typing
import Iterator, Optional\n\nclass ProcessTableOperator(UDFTableOperator):\n\n
@overrides\n def process_table(self, table: Table, port: int) ->
Iterator[Optional[TableLike]]:\n X_train = table['X_train'].iloc[0]\n
y_train = table['y_train'].iloc[0]\n X_test =
table['X_test'].iloc[0]\n y_test = table['y_test'].iloc[0]\n\n #
Train SVM model\n svm_model = SVC(random_state=42)\n
svm_model.fit(X_train, y_train)\n svm_pred = svm_model.predict(X_test)\n
svm_accuracy = accuracy_score(y_test, svm_pred)\n\n # Create a
DataFrame to yield\n result_table = pd.DataFrame({\n
'svm_model': [svm_model],\n 'svm_accuracy': [svm_accuracy],\n
'X_test': [X_test],\n 'y_test': [y_test]\n })\n\n
yield Table(result_tab
le)"
+ },
+ "edges": [
+ ["UDF1", "UDF2"],
+ ["UDF1", "UDF3"],
+ ["UDF3", "UDF4"],
+ ["UDF3", "UDF5"]
+ ],
+ "outputs": {
+ "UDF1": ["min_values", "max_values", "mean_values", "data"],
+ "UDF2": ["html-content"],
+ "UDF3": ["X_train", "X_test", "y_train", "y_test"],
+ "UDF4": ["rf_model", "rf_accuracy", "X_test", "y_test"],
+ "UDF5": ["svm_model", "svm_accuracy", "X_test", "y_test"]
+ }
+}
+\`\`\``;
+
+export const SCRIPT_WORKFLOW_PROMPT = `You are an expert in Python coding and
workflow systems.
+Many users of Texera system are non-technical, but the Python scripts they
provide are written by technical people.
+They want to convert their scripts to Texera workflows.
+Your goal is to help convert these scripts into a Texera workflow that
non-technical users can use directly.
+So do not remove or modify any classes or functions, preserve their names and
structure as they are.
+Ensure that all essential logic remains intact.
+Create multiple Texera UDF codes using the provided Python code.
+Number each UDF, starting at 1 and incrementing, by starting with a comment
that states that UDF number.
+
+Use the class and function names as shown in ProcessTupleOperator,
ProcessTableOperator, and ProcessBatchOperator.
+Do not change the class names, function names, or input parameters.
+Use the ones that make sense and split the code meaningfully as instructed.
+
+Use the starter code provided for Python UDFs.
+
+Use the documentation of Table, Tuple, or Batch to work with parameters within
Texera UDF.
+Do not import other libraries to define these types.
+
+There is no need for an __init__ function. Assume all inputs are valid pandas
DataFrames,
+so do not use .to_pandas(), .to_dataframe(), etc. Do not load data from a file
in the first UDF;
+the workflow's source operator supplies the initial data, so assume it is
already given to you in the
+table parameter. Replacing file-loading code with this input is the one
exception to preserving all
+original code (see below).
+Ensure proper data flow between functions. Separate operators as if they will
run in different files.
+
+Current UDF operators can only have one output. Build a dataframe to yield all
necessary variables
+and data. Ensure proper data flow for each UDF and all information is yielded
(including training
+and testing data) if subsequent UDFs need them.
+
+Ensure all necessary imports are included in each UDF code block.
+
+Each UDF operator should be in its own Python code block. Do not combine them
into a single block.
+Ensure import statements cover all used functions and separate them as
necessary.
+
+It is VERY important that all of the original code in the Python script is
represented in the generated workflow.
+Make sure that nothing in the original is removed and that the semantic
meaning of what the original code was doing is retained.
+The only exception is data-loading code (e.g. pd.read_csv); it is represented
by the workflow's input/source operator rather than copied into a UDF.
+If there are user-defined Python classes, include the entire class definition
in the appropriate UDF(s) that use that class.
+Always include the code that defines the class inside of every distinct UDF
that uses that constructs an object of that class.
Review Comment:
This sentence is grammatically malformed and makes the model instruction
ambiguous. State directly that each UDF constructing the class must include its
definition.
##########
frontend/src/app/workspace/service/notebook-migration/migration-prompts.ts:
##########
@@ -412,3 +412,173 @@ Here is an example of a mapping generated between the
given example Python code
}
Now create a mapping for the UDFs and the original code. Link the code blocks
marked by 'START <cell-uuid>' and 'END <cell-uuid>' with the UDF UUID's. The
code between them should be equivalent. Multiple cells can be mapped to the
same UDF when that UDF implements the logic of those cells. There could be any
number of cells and UDFs, so only create the correct number in the mapping.
Only give the mapping.
`;
+
+export const EXAMPLE_OF_MULTIPLE_UDF_CONVERSION_SCRIPT = `
+Here is an example of breaking up python code into multiple Texera UDFs.
Format your response structure exactly like the given example. The "code" key
contains a dictionary of the UDF ID's with their respective code. The "edges"
key contains a list of pairs that contains the connections between UDFs. The
"outputs" key contains a dictionary of the UDF ID's with a list of the output
column names of the DataFrame that the UDF yields. The UDFs can branch and
merge, it does not have to be a linear chain depending on your implementation.
+
+The original code is shown with each line prefixed by its line number and a
'|'. Those prefixes are annotations so that line ranges can be referred to
later. They are not part of the code and must never appear in the code you
generate.
+
+Original Code:
+\`\`\`python
+ 1| import pandas as pd
+ 2| from sklearn.model_selection import train_test_split
+ 3| from sklearn.ensemble import RandomForestClassifier
+ 4| from sklearn.svm import SVC
+ 5| from sklearn.tree import DecisionTreeClassifier
+ 6| from sklearn.linear_model import LogisticRegression
+ 7| from sklearn.metrics import accuracy_score
+ 8| from sklearn.preprocessing import StandardScaler
+ 9| import matplotlib.pyplot as plt
+10|
+11| # Load the dataset
+12| file_path = 'diabetes.csv'
+13| data = pd.read_csv(file_path)
+14|
+15| # Remove duplicate rows
+16| data = data.drop_duplicates()
+17|
+18| # Remove rows with null values
+19| data = data.dropna()
+20|
+21| # Print the minimum, maximum, and mean for all fields
+22| print("Minimum values:", data.min())
+23| print("Maximum values:", data.max())
+24| print("Mean values:", data.mean())
+25|
+26| # Create a boxplot for the 'Pregnancies' field
+27| plt.figure(figsize=(8, 6))
+28| plt.boxplot(data['Pregnancies'], vert=False, patch_artist=True)
+29| plt.title('Boxplot of Pregnancies')
+30| plt.xlabel('Number of Pregnancies')
+31| plt.show()
+32|
+33| # Separate features and target variable
+34| X = data.drop('Outcome', axis=1)
+35| y = data['Outcome']
+36|
+37| # Split data into training and testing sets (80% train, 20% test)
+38| X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=42)
+39|
+40| scaler = StandardScaler()
+41| X_train = scaler.fit_transform(X_train)
+42| X_test = scaler.transform(X_test)
+43|
+44| # Train Random Forest model
+45| rf_model = RandomForestClassifier(random_state=42)
+46| rf_model.fit(X_train, y_train)
+47| rf_pred = rf_model.predict(X_test)
+48| rf_accuracy = accuracy_score(y_test, rf_pred)
+49| print(f"Random Forest Accuracy: {rf_accuracy:.2%}")
+50|
+51| # Train SVM model
+52| svm_model = SVC(random_state=42)
+53| svm_model.fit(X_train, y_train)
+54| svm_pred = svm_model.predict(X_test)
+55| svm_accuracy = accuracy_score(y_test, svm_pred)
+56| print(f"SVM Accuracy: {svm_accuracy:.2%}")
+\`\`\`
+
+Texera UDF conversion:
+\`\`\`json
+{
+ "code": {
+ "UDF1": "# UDF1\nfrom pytexera import *\nimport pandas as pd\nfrom
typing import Iterator, Optional\n\nclass
ProcessTableOperator(UDFTableOperator):\n\n @overrides\n def
process_table(self, table: Table, port: int) ->
Iterator[Optional[TableLike]]:\n # Remove duplicate rows\n data =
table.drop_duplicates()\n\n # Remove rows with null values\n data
= data.dropna()\n\n # Calculate statistics\n min_values =
data.min()\n max_values = data.max()\n mean_values =
data.mean()\n\n # Create a DataFrame to yield\n result_table =
pd.DataFrame({\n 'min_values': [min_values],\n
'max_values': [max_values],\n 'mean_values': [mean_values],\n
'data': [data]\n })\n\n yield Table(result_table)",
+ "UDF2": "# UDF2\nfrom pytexera import *\nimport pandas as pd\nimport
plotly.express as px\nimport plotly.io\nfrom typing import Iterator,
Optional\n\nclass ProcessTableOperator(UDFTableOperator):\n def
render_error(self, error_msg):\n return '''<h1>Boxplot is not
available.</h1>\n <p>Reason is: {} </p>\n
'''.format(error_msg)\n\n @overrides\n def process_table(self, table:
Table, port: int) -> Iterator[Optional[TableLike]]:\n data =
table['data'].iloc[0]\n\n if data.empty:\n yield
{'html-content': self.render_error('input table is empty.')}\n
return\n\n # Create a boxplot for the 'Pregnancies' field\n fig =
px.box(data, x='Pregnancies')\n fig.update_layout(margin=dict(l=0, r=0,
t=0, b=0))\n\n # Convert fig to HTML content\n html =
plotly.io.to_html(fig, include_plotlyjs='cdn', auto_play=False)\n yield
{'html-content': html}",
+ "UDF3": "# UDF3\nfrom pytexera import *\nimport pandas as pd\nfrom
sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing
import StandardScaler\nfrom typing import Iterator, Optional\n\nclass
ProcessTableOperator(UDFTableOperator):\n\n @overrides\n def
process_table(self, table: Table, port: int) ->
Iterator[Optional[TableLike]]:\n data = table['data'].iloc[0]\n\n
# Separate features and target variable\n X = data.drop('Outcome',
axis=1)\n y = data['Outcome']\n\n # Split data into training and
testing sets (80% train, 20% test)\n X_train, X_test, y_train, y_test =
train_test_split(X, y, test_size=0.2, random_state=42)\n\n scaler =
StandardScaler()\n X_train = scaler.fit_transform(X_train)\n
X_test = scaler.transform(X_test)\n\n # Create a DataFrame to yield\n
result_table = pd.DataFrame({\n 'X_train': [X_train], 'X_test':
[X_test],\n 'y_train': [y_tra
in], 'y_test': [y_test]\n })\n\n yield Table(result_table)",
+ "UDF4": "# UDF4\nfrom pytexera import *\nimport pandas as pd\nfrom
sklearn.ensemble import RandomForestClassifier\nfrom sklearn.metrics import
accuracy_score\nfrom typing import Iterator, Optional\n\nclass
ProcessTableOperator(UDFTableOperator):\n\n @overrides\n def
process_table(self, table: Table, port: int) ->
Iterator[Optional[TableLike]]:\n X_train = table['X_train'].iloc[0]\n
y_train = table['y_train'].iloc[0]\n X_test =
table['X_test'].iloc[0]\n y_test = table['y_test'].iloc[0]\n\n #
Train Random Forest model\n rf_model =
RandomForestClassifier(random_state=42)\n rf_model.fit(X_train,
y_train)\n rf_pred = rf_model.predict(X_test)\n rf_accuracy =
accuracy_score(y_test, rf_pred)\n\n # Create a DataFrame to yield\n
result_table = pd.DataFrame({\n 'rf_model': [rf_model],\n
'rf_accuracy': [rf_accuracy],\n 'X_test': [X_test],\n
'y_test': [y_test]\n
})\n\n yield Table(result_table)",
+ "UDF5": "# UDF5\nfrom pytexera import *\nimport pandas as pd\nfrom
sklearn.svm import SVC\nfrom sklearn.metrics import accuracy_score\nfrom typing
import Iterator, Optional\n\nclass ProcessTableOperator(UDFTableOperator):\n\n
@overrides\n def process_table(self, table: Table, port: int) ->
Iterator[Optional[TableLike]]:\n X_train = table['X_train'].iloc[0]\n
y_train = table['y_train'].iloc[0]\n X_test =
table['X_test'].iloc[0]\n y_test = table['y_test'].iloc[0]\n\n #
Train SVM model\n svm_model = SVC(random_state=42)\n
svm_model.fit(X_train, y_train)\n svm_pred = svm_model.predict(X_test)\n
svm_accuracy = accuracy_score(y_test, svm_pred)\n\n # Create a
DataFrame to yield\n result_table = pd.DataFrame({\n
'svm_model': [svm_model],\n 'svm_accuracy': [svm_accuracy],\n
'X_test': [X_test],\n 'y_test': [y_test]\n })\n\n
yield Table(result_tab
le)"
+ },
+ "edges": [
+ ["UDF1", "UDF2"],
+ ["UDF1", "UDF3"],
+ ["UDF3", "UDF4"],
+ ["UDF3", "UDF5"]
+ ],
+ "outputs": {
+ "UDF1": ["min_values", "max_values", "mean_values", "data"],
+ "UDF2": ["html-content"],
+ "UDF3": ["X_train", "X_test", "y_train", "y_test"],
+ "UDF4": ["rf_model", "rf_accuracy", "X_test", "y_test"],
+ "UDF5": ["svm_model", "svm_accuracy", "X_test", "y_test"]
+ }
+}
+\`\`\``;
+
+export const SCRIPT_WORKFLOW_PROMPT = `You are an expert in Python coding and
workflow systems.
+Many users of Texera system are non-technical, but the Python scripts they
provide are written by technical people.
+They want to convert their scripts to Texera workflows.
+Your goal is to help convert these scripts into a Texera workflow that
non-technical users can use directly.
+So do not remove or modify any classes or functions, preserve their names and
structure as they are.
+Ensure that all essential logic remains intact.
+Create multiple Texera UDF codes using the provided Python code.
+Number each UDF, starting at 1 and incrementing, by starting with a comment
that states that UDF number.
+
+Use the class and function names as shown in ProcessTupleOperator,
ProcessTableOperator, and ProcessBatchOperator.
+Do not change the class names, function names, or input parameters.
+Use the ones that make sense and split the code meaningfully as instructed.
+
+Use the starter code provided for Python UDFs.
+
+Use the documentation of Table, Tuple, or Batch to work with parameters within
Texera UDF.
+Do not import other libraries to define these types.
+
+There is no need for an __init__ function. Assume all inputs are valid pandas
DataFrames,
+so do not use .to_pandas(), .to_dataframe(), etc. Do not load data from a file
in the first UDF;
+the workflow's source operator supplies the initial data, so assume it is
already given to you in the
+table parameter. Replacing file-loading code with this input is the one
exception to preserving all
+original code (see below).
+Ensure proper data flow between functions. Separate operators as if they will
run in different files.
+
+Current UDF operators can only have one output. Build a dataframe to yield all
necessary variables
+and data. Ensure proper data flow for each UDF and all information is yielded
(including training
+and testing data) if subsequent UDFs need them.
+
+Ensure all necessary imports are included in each UDF code block.
+
+Each UDF operator should be in its own Python code block. Do not combine them
into a single block.
+Ensure import statements cover all used functions and separate them as
necessary.
+
+It is VERY important that all of the original code in the Python script is
represented in the generated workflow.
+Make sure that nothing in the original is removed and that the semantic
meaning of what the original code was doing is retained.
+The only exception is data-loading code (e.g. pd.read_csv); it is represented
by the workflow's input/source operator rather than copied into a UDF.
+If there are user-defined Python classes, include the entire class definition
in the appropriate UDF(s) that use that class.
+Always include the code that defines the class inside of every distinct UDF
that uses that constructs an object of that class.
+Python classes are allowed in Texera UDFs and follow the same semantics as
standard Python.
+They can be defined outside of ProcessTableOperator, ProcessTupleOperator, and
ProcessBatchOperator.
+
+Return only the JSON formatted response, do not give any explanation.
+Do not wrap the JSON in markdown code fences. Output raw JSON only.
+Make sure the response is a valid JSON structure, including closing all braces
and not including commas after the last element.
+Follow this JSON format (don't reuse the values, this is just the format).
'code', 'edges', and 'outputs' are all their own key's, do not nest any of
these in another one and make sure to close their braces:
+{
+"code": {
+"UDF1": "code for UDF1 goes here",
+"UDF2": "code for UDF2 goes here"
+},
+"edges": [
+["UDF1", "UDF2"]
+],
+"outputs": {
+"UDF1": ["min_values", "max_values", "mean_values", "data"],
+"UDF2": ["html-content"]
+}
+}
+Make sure only the keys in the code section appear in the edges and outputs
sections. Do not include any extraneous fields.
+Do not include any extraneous UDF's in the code field that include empty
strings.
+Give ALL of the code, do not omit anything or use placeholders for code. Make
sure ALL code in the original is translated over.
+The value of each UDF must be a valid JSON string: escape newlines, quotes,
and backslashes correctly so that the decoded string is runnable Python. Use
whichever quotes the Python code requires.
+Each line of the script below is prefixed with its line number followed by '|
'. Those prefixes are annotations
+so that line ranges can be referred to later; never reproduce them in any
generated UDF code.
+Convert following the instructions and examples given. Here is the code:
+`;
+
+export const SCRIPT_MAPPING_PROMPT = `
+Here is an example of a mapping generated between the given example Python
code and the Texera UDFs, using line ranges of the original script and the UDF
IDs. A range is a pair [firstLine, lastLine]; both bounds are 1-indexed and
inclusive, and they refer to the line numbers shown in the prefix of the
original code. A UDF may list several ranges when its logic came from separate
parts of the script. The format should be kept the same.
+{
+"UDF1": [[15, 24]],
+"UDF2": [[26, 31]],
+"UDF3": [[33, 42]],
+"UDF4": [[44, 49]]
Review Comment:
This worked example defines five UDFs, but the mapping omits UDF5 even
though lines 51–56 implement its SVM logic. Because this example is used to
teach the model the response shape, it encourages incomplete mappings and can
leave the generated SVM operator without highlighting. Include UDF5 in the
example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
