This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-7645-604f10967b368a5a34ae41649e5ecc915f2c9a5c in repository https://gitbox.apache.org/repos/asf/texera.git
commit 8cccfd8b438278ce97b8b73327f7dd81d37e1d46 Author: Kary Zheng <[email protected]> AuthorDate: Sat Aug 29 04:41:37 2026 +0000 feat(workflow-operator): constrain the Sklearn text columns, and drop what an estimator cannot fit (#7645) ### What changes were proposed in this PR? Two configurations ended the run with an error from inside scikit-learn, naming neither the column nor the field to change. The text columns now state what they take. `Count Vectorizer` tokenizes documents, so those columns are strings, and at least one is required exactly when that switch is on: with the switch off nothing reads them, and with the switch on an empty list reaches scikit-learn as a `ColumnTransformer` holding no transformers, which fits an array of no features and raises `ValueError: Found array with 0 feature(s)`. Both are schema constraints, the second in the conditional form Aggregate already uses, so the panel refuses the configuration while it is being written. Conditional rather than a plain `required`, so a freshly dropped operator, whose vectorizer is off, is not flagged for a field it has no use for. The feature set drops what it cannot fit rather than ending the run. These operators take every column but the target, so a text column beside the numbers, one the user never meant as a feature, raised `ValueError: could not convert string to float`, and a timestamp raised `DTypePromotionError`. Nothing in the configuration could exclude it, and the message named neither the column nor a way out. Booleans are kept, fitting as 0/1. What was left out is printed, so the choice is visible rather than silent, and this follows the rest of the codebase, where twenty-four visualization operators drop missing values before plotting. A table whose every column but the target is text is narrowed down to nothing, and a frame of no columns has no dtype for scikit-learn to read, so numpy raised `at least one array or dtype is required`, naming neither the columns nor a way out. The narrowing says it itself instead, naming what it left out and asking for a numeric or boolean column. The drop is what makes this reachable, so it belongs with it rather than in a change of its own. The drop is skipped under the text pipeline, where the `ColumnTransformer` names the columns it reads and they are exactly the ones this would drop. The schema constraints sit on the shared base, so they reach all fifty-one operators of the Sklearn and Sklearn Training groups. The required marker for the text columns needs #7642; without it the configuration is still refused, only unmarked. The narrowing reaches three more operators, all of which take every column but the target and descend from `PythonOperatorDescriptor` rather than one of the sklearn bases. Linear Regression ended the run on a text column the way the fifty-one used to. Prediction and Testing needed it for a second reason: a model is fitted by one operator and read by another, so once the fitting side leaves a column out, the reading side has to leave out the same one or scikit-learn refuses the frame for naming a feature it never saw. Prediction reads `feature_names_in_` off the model rather than re-deriving the rule, since its generated code holds one Tuple rather than a frame. The one definition now lives in a trait mixed in wherever a frame is handed to an estimator, so a fourth operator cannot quietly disagree with the other three. This is rebased on #7668, which rewrote the same line of the two codegen templates. That line is resolved the way #7668 asked: the branch this PR introduces is kept, the one #7668 removed stays removed, and the condition moved into `dropNonFeatureColumns`, which now emits nothing when `Count Vectorizer` is on. ### Any related issues, documentation, discussions? Closes #7643. The marker for the new conditional required comes from #7642. ### How was this PR tested? `SklearnClassifierOpDescCodegenSpec` and `SklearnTrainingOpDescCodegenSpec` already pinned the generated feature-selection line and were updated to the new one, including that the text pipeline does not carry it. The three operators added here pin theirs in their own specs, and removing a narrowing turns the matching one red. The message for a narrowing that leaves no column is pinned in `SklearnLinearRegressionOpDescSpec`, which already owns the assertions on that generated block. `WorkflowOperator/test` passes: 2456 tests. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]> --- .../operator/sklearn/SklearnClassifierOpDesc.scala | 1 + .../operator/sklearn/SklearnFittableColumns.scala | 61 ++++++++++++++++++++++ .../sklearn/SklearnLinearRegressionOpDesc.scala | 3 +- .../operator/sklearn/SklearnModelOpDesc.scala | 38 +++++++++++++- .../operator/sklearn/SklearnPredictionOpDesc.scala | 3 ++ .../sklearn/testing/SklearnTestingOpDesc.scala | 4 +- .../sklearn/training/SklearnTrainingOpDesc.scala | 1 + .../SklearnClassifierOpDescCodegenSpec.scala | 11 +++- .../SklearnLinearRegressionOpDescSpec.scala | 21 ++++++++ .../sklearn/SklearnPredictionOpDescSpec.scala | 16 ++++++ .../sklearn/testing/SklearnTestingOpDescSpec.scala | 13 +++++ .../SklearnTrainingOpDescCodegenSpec.scala | 11 +++- 12 files changed, 176 insertions(+), 7 deletions(-) diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDesc.scala index 588d717728..a2323e5d19 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDesc.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDesc.scala @@ -46,6 +46,7 @@ abstract class SklearnClassifierOpDesc extends SklearnModelOpDesc { | print("Skipped", rows_read - len(table), "of", rows_read, "rows with missing values") | Y = table[$target] | X = table.drop($target, axis=1) + |${dropNonFeatureColumns("X", " " * 8)} $reportMissingKept | if port == 0: | self.model = make_pipeline(${vectorizerStage(c => pyb"$c".toString)} ${if ( diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnFittableColumns.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnFittableColumns.scala new file mode 100644 index 0000000000..97e3cd1d7d --- /dev/null +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnFittableColumns.scala @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.amber.operator.sklearn + +import com.fasterxml.jackson.annotation.JsonIgnore + +/** The one definition of which columns an estimator is given. + * + * A model is fitted by one operator and read by another, so the rule has to + * hold across operators rather than within one: a column the fitting side + * leaves out is a column the predicting side must leave out too, or scikit-learn + * refuses the frame for naming features it never saw. One definition, mixed in + * wherever a frame is handed to an estimator, is what makes that true by + * construction instead of by four copies agreeing. + * + * Not on [[SklearnModelOpDesc]], where this began: Linear Regression and the + * prediction and testing operators descend from `PythonOperatorDescriptor` + * directly and could not reach it there. + */ +trait SklearnFittableColumns { + + /** Python that narrows `frame` to the columns an estimator can fit, written at + * `indent`. + * + * A column the user did not mean as a feature, a note beside the numbers, + * would otherwise end the run from inside scikit-learn. Booleans are kept: + * they fit as 0/1. What was dropped is printed, so the choice is visible + * rather than silent. + * + * A table whose every column but the target is text leaves nothing behind, + * and a frame of no columns has no dtype for scikit-learn to read, so it + * raises `at least one array or dtype is required` from inside numpy. The + * narrowing says so itself instead, naming the columns it left out. + */ + @JsonIgnore + protected def narrowToFittableColumns(frame: String, indent: String): String = + s"""${indent}_fittable = $frame.select_dtypes(include=["number", "bool"]) + |${indent}_ignored = [c for c in $frame.columns if c not in _fittable.columns] + |${indent}if _ignored: + |${indent} print("Ignoring columns an estimator cannot fit:", _ignored) + |${indent}if _fittable.columns.empty: + |${indent} raise ValueError(f"No column left to fit on: an estimator cannot fit {_ignored}. Give it a numeric or boolean column.") + |${indent}$frame = _fittable""".stripMargin +} diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDesc.scala index e590fa6928..9ea793338e 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDesc.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDesc.scala @@ -29,7 +29,7 @@ import org.apache.texera.amber.operator.PythonOperatorDescriptor import org.apache.texera.amber.operator.metadata.annotations.AutofillAttributeName import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo} -class SklearnLinearRegressionOpDesc extends PythonOperatorDescriptor { +class SklearnLinearRegressionOpDesc extends PythonOperatorDescriptor with SklearnFittableColumns { @JsonSchemaTitle("Target Attribute") @JsonPropertyDescription("Attribute in your dataset corresponding to target.") @@ -59,6 +59,7 @@ class SklearnLinearRegressionOpDesc extends PythonOperatorDescriptor { | print("Skipped", rows_read - len(table), "of", rows_read, "rows with missing values") | Y = table[$target] | X = table.drop($target, axis=1) + |${narrowToFittableColumns("X", " " * 8)} | if port == 0: | pipeline = make_pipeline( | PolynomialFeatures(degree=$degree), diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnModelOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnModelOpDesc.scala index 74e474f1ac..c771d31aff 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnModelOpDesc.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnModelOpDesc.scala @@ -42,7 +42,35 @@ import org.apache.texera.amber.operator.metadata.annotations.{ HideAnnotation } -abstract class SklearnModelOpDesc extends PythonOperatorDescriptor { +// `text` names the columns Count Vectorizer tokenizes, so they are string columns +// and at least one is required only when that switch is on. Conditional rather than +// plain required, so a freshly dropped operator is not flagged for a field it has +// no use for. +@JsonSchemaInject(json = """ +{ + "attributeTypeRules": { + "text": { + "enum": ["string"] + } + }, + "allOf": [ + { + "if": { + "properties": { + "countVectorizer": { "const": true } + } + }, + "then": { + "required": ["text"], + "properties": { + "text": { "minItems": 1 } + } + } + } + ] +} +""") +abstract class SklearnModelOpDesc extends PythonOperatorDescriptor with SklearnFittableColumns { @JsonSchemaTitle("Target Attribute") @JsonPropertyDescription("Attribute in your dataset corresponding to target.") @@ -112,6 +140,14 @@ abstract class SklearnModelOpDesc extends PythonOperatorDescriptor { .map { case (column, i) => s"""("text$i", CountVectorizer(), ${renderColumn(column)})""" } .mkString("ColumnTransformer([", ", ", "]),") + /** [[SklearnFittableColumns.narrowToFittableColumns]], except under the text + * pipeline: there the `ColumnTransformer` names the columns it reads, and they + * are the ones the narrowing would drop. + */ + @JsonIgnore + protected def dropNonFeatureColumns(frame: String, indent: String): String = + if (countVectorizer) "" else narrowToFittableColumns(frame, indent) + @JsonIgnore def getImportStatements: String diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDesc.scala index 0957339ab4..5d8b1b1e7d 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDesc.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDesc.scala @@ -62,6 +62,9 @@ class SklearnPredictionOpDesc extends PythonOperatorDescriptor { | input_features = tuple_ | if $groundTruthAttribute != "": | input_features = input_features.get_partial_tuple([col for col in tuple_.get_field_names() if col != $groundTruthAttribute]) + | _fitted = getattr(self.model, "feature_names_in_", None) + | if _fitted is not None: + | input_features = input_features.get_partial_tuple(list(_fitted)) | if Table.from_tuple_likes([input_features]).isna().any(axis=None): | tuple_[$resultAttribute] = None #keep the row, leave the result empty | else: diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDesc.scala index 8e633169ee..94319f0936 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDesc.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDesc.scala @@ -24,6 +24,7 @@ import com.kjetland.jackson.jsonSchema.annotations.JsonSchemaTitle import org.apache.texera.amber.core.tuple.{AttributeType, Schema} import org.apache.texera.amber.core.workflow.{InputPort, OutputPort, PortIdentity} import org.apache.texera.amber.operator.PythonOperatorDescriptor +import org.apache.texera.amber.operator.sklearn.SklearnFittableColumns import org.apache.texera.amber.operator.metadata.annotations.{ AutofillAttributeName, AutofillAttributeNameOnPort1 @@ -32,7 +33,7 @@ import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, Operat import org.apache.texera.amber.pybuilder.PyStringTypes.EncodableString import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.PythonTemplateBuilderStringContext -class SklearnTestingOpDesc extends PythonOperatorDescriptor { +class SklearnTestingOpDesc extends PythonOperatorDescriptor with SklearnFittableColumns { @JsonProperty(required = true, defaultValue = "false") @JsonSchemaTitle("Regression") @JsonPropertyDescription( @@ -74,6 +75,7 @@ class SklearnTestingOpDesc extends PythonOperatorDescriptor { | print("Skipped", rows_read - len(table), "of", rows_read, "rows with missing values") | Y = table[$target] | X = table.drop($target, axis=1) + |${narrowToFittableColumns("X", " " * 12)} | predictions = model.predict(X.squeeze()) | if $isRegressionStr: | tuple_["R2"] = r2_score(Y, predictions) diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/training/SklearnTrainingOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/training/SklearnTrainingOpDesc.scala index 3de809dd9f..c51f322a62 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/training/SklearnTrainingOpDesc.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/training/SklearnTrainingOpDesc.scala @@ -46,6 +46,7 @@ class SklearnTrainingOpDesc extends SklearnModelOpDesc { | print("Skipped", rows_read - len(table), "of", rows_read, "rows with missing values") | Y = table[$target] | X = table.drop($target, axis=1) + |${dropNonFeatureColumns("X", " " * 8)} $reportMissingKept | model = make_pipeline(${vectorizerStage(c => pyb"$c".toString)} ${if ( tfidfTransformer diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDescCodegenSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDescCodegenSpec.scala index e964fcca34..0bdd32ec90 100644 --- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDescCodegenSpec.scala +++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDescCodegenSpec.scala @@ -68,8 +68,12 @@ class SklearnClassifierOpDescCodegenSpec extends AnyFlatSpec with Matchers { code should include("from sklearn.neighbors import KNeighborsClassifier") code should include(s"Y = table[${decodeExpr("label")}]") code should include(s"X = table.drop(${decodeExpr("label")}, axis=1)") - // Feature-column path: X is kept whole, the text attribute is never read. + // Feature-column path: every column an estimator can fit is kept, the rest are + // named on the console, and the text attribute is never read. code should not include "ColumnTransformer(" + code should include("""_fittable = X.select_dtypes(include=["number", "bool"])""") + code should include("""print("Ignoring columns an estimator cannot fit:", _ignored)""") + code should include("X = _fittable") code should not include decodeExpr("docs") normalized(code) should include( "self.model = make_pipeline( KNeighborsClassifier()).fit(X, Y)" @@ -81,7 +85,9 @@ class SklearnClassifierOpDescCodegenSpec extends AnyFlatSpec with Matchers { it should "select the text column and prepend CountVectorizer when countVectorizer is on" in { val code = descriptor(countVectorizer = true).generatePythonCode() - // ColumnTransformer selects the columns itself, so X stays the whole frame. + // ColumnTransformer selects the columns itself, so X stays the whole frame, and + // narrowing it to the fittable columns would drop the text ones it reads. + code should not include "_fittable" normalized(code) should include( s"""self.model = make_pipeline(ColumnTransformer([("text0", CountVectorizer(), ${decodeExpr( "docs" @@ -120,6 +126,7 @@ class SklearnClassifierOpDescCodegenSpec extends AnyFlatSpec with Matchers { val code = descriptor(tfidfTransformer = true).generatePythonCode() // Without countVectorizer there is no text-column selection. code should not include "ColumnTransformer(" + code should include("X = _fittable") code should not include decodeExpr("docs") normalized(code) should include( "self.model = make_pipeline( TfidfTransformer(), KNeighborsClassifier()).fit(X, Y)" diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDescSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDescSpec.scala index 2c1c02a9eb..d0364e1998 100644 --- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDescSpec.scala +++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDescSpec.scala @@ -74,6 +74,27 @@ class SklearnLinearRegressionOpDescSpec extends AnyFlatSpec with Matchers { code should include("\"Skipped\"") } + // The operator fits every column but the target, so a text column beside the + // numbers would end the run from inside scikit-learn. + it should "narrow the features to the columns an estimator can fit" in { + val d = new SklearnLinearRegressionOpDesc + d.target = "y" + val code = d.generatePythonCode() + code should include("""_fittable = X.select_dtypes(include=["number", "bool"])""") + code should include("""print("Ignoring columns an estimator cannot fit:", _ignored)""") + code should include("X = _fittable") + } + + // Narrowing a table whose every column but the target is text leaves a frame of + // no columns, which numpy answers with `at least one array or dtype is required`. + it should "say so when the narrowing leaves no column at all" in { + val d = new SklearnLinearRegressionOpDesc + d.target = "y" + val code = d.generatePythonCode() + code should include("if _fittable.columns.empty:") + code should include("No column left to fit on") + } + "SklearnLinearRegressionOpDesc" should "round-trip its target through the polymorphic base" in { val d = new SklearnLinearRegressionOpDesc diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDescSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDescSpec.scala index 61e6b1750e..16aec5da13 100644 --- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDescSpec.scala +++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDescSpec.scala @@ -138,6 +138,22 @@ class SklearnPredictionOpDescSpec extends AnyFlatSpec with Matchers { code should include("str(prediction)") } + // The fitting operators leave out the columns an estimator cannot fit, so this + // side has to leave out the same ones or scikit-learn refuses the frame for + // naming features it never saw. Read off the model, which carries what it was + // fitted on, rather than re-deriving a rule that could drift from theirs: this + // path holds one Tuple rather than a frame, where select_dtypes does not apply. + it should "narrow the input features to the ones the model was fitted on" in { + val d = new SklearnPredictionOpDesc + d.model = "model" + d.resultAttribute = "prediction" + d.groundTruthAttribute = "y" + val code = d.generatePythonCode() + code should include(""""feature_names_in_", None)""") + code should include("if _fitted is not None:") + code should include("input_features.get_partial_tuple(list(_fitted))") + } + "SklearnPredictionOpDesc" should "round-trip its config fields through the polymorphic base" in { val d = new SklearnPredictionOpDesc diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDescSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDescSpec.scala index 68ec31a8d7..8401c6e20b 100644 --- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDescSpec.scala +++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDescSpec.scala @@ -89,6 +89,19 @@ class SklearnTestingOpDescSpec extends AnyFlatSpec with Matchers { d.generatePythonCode() should include("Table(self.data).dropna()") } + // The scorer reads every column but the target, so it has to leave out what an + // estimator cannot fit for the same reason the fitting operators do, and leave + // out the same columns: a model fitted without them refuses a frame naming them. + it should "narrow the features to the columns an estimator can fit" in { + val d = new SklearnTestingOpDesc + d.model = "model" + d.target = "y" + val code = d.generatePythonCode() + code should include("""_fittable = X.select_dtypes(include=["number", "bool"])""") + code should include("""print("Ignoring columns an estimator cannot fit:", _ignored)""") + code should include("X = _fittable") + } + "SklearnTestingOpDesc" should "round-trip its config fields through the polymorphic base" in { val d = new SklearnTestingOpDesc diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/training/SklearnTrainingOpDescCodegenSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/training/SklearnTrainingOpDescCodegenSpec.scala index 114393d374..23301e18c7 100644 --- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/training/SklearnTrainingOpDescCodegenSpec.scala +++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/training/SklearnTrainingOpDescCodegenSpec.scala @@ -68,8 +68,12 @@ class SklearnTrainingOpDescCodegenSpec extends AnyFlatSpec with Matchers { code should include("from sklearn.neighbors import KNeighborsClassifier") code should include(s"Y = table[${decodeExpr("label")}]") code should include(s"X = table.drop(${decodeExpr("label")}, axis=1)") - // Feature-column path: X is kept whole, the text attribute is never read. + // Feature-column path: every column an estimator can fit is kept, the rest are + // named on the console, and the text attribute is never read. code should not include "ColumnTransformer(" + code should include("""_fittable = X.select_dtypes(include=["number", "bool"])""") + code should include("""print("Ignoring columns an estimator cannot fit:", _ignored)""") + code should include("X = _fittable") code should not include decodeExpr("docs") normalized(code) should include("make_pipeline( KNeighborsClassifier()).fit(X, Y)") code should not include "CountVectorizer()" @@ -78,7 +82,9 @@ class SklearnTrainingOpDescCodegenSpec extends AnyFlatSpec with Matchers { it should "select the text column and prepend CountVectorizer when countVectorizer is on" in { val code = descriptor(countVectorizer = true).generatePythonCode() - // ColumnTransformer selects the columns itself, so X stays the whole frame. + // ColumnTransformer selects the columns itself, so X stays the whole frame, and + // narrowing it to the fittable columns would drop the text ones it reads. + code should not include "_fittable" normalized(code) should include( s"""make_pipeline(ColumnTransformer([("text0", CountVectorizer(), ${decodeExpr( "docs" @@ -117,6 +123,7 @@ class SklearnTrainingOpDescCodegenSpec extends AnyFlatSpec with Matchers { val code = descriptor(tfidfTransformer = true).generatePythonCode() // Without countVectorizer there is no text-column selection. code should not include "ColumnTransformer(" + code should include("X = _fittable") code should not include decodeExpr("docs") normalized(code) should include( "make_pipeline( TfidfTransformer(), KNeighborsClassifier()).fit(X, Y)"
