This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/texera.git
The following commit(s) were added to refs/heads/main by this push:
new 8cccfd8b43 feat(workflow-operator): constrain the Sklearn text
columns, and drop what an estimator cannot fit (#7645)
8cccfd8b43 is described below
commit 8cccfd8b438278ce97b8b73327f7dd81d37e1d46
Author: Kary Zheng <[email protected]>
AuthorDate: Sat Aug 29 04:41:37 2026 +0000
feat(workflow-operator): constrain the Sklearn text columns, and drop what
an estimator cannot fit (#7645)
### What changes were proposed in this PR?
Two configurations ended the run with an error from inside scikit-learn,
naming neither the column nor the field to change.
The text columns now state what they take. `Count Vectorizer` tokenizes
documents, so those columns are strings, and at least one is required
exactly when that switch is on: with the switch off nothing reads them,
and with the switch on an empty list reaches scikit-learn as a
`ColumnTransformer` holding no transformers, which fits an array of no
features and raises `ValueError: Found array with 0 feature(s)`. Both
are schema constraints, the second in the conditional form Aggregate
already uses, so the panel refuses the configuration while it is being
written. Conditional rather than a plain `required`, so a freshly
dropped operator, whose vectorizer is off, is not flagged for a field it
has no use for.
The feature set drops what it cannot fit rather than ending the run.
These operators take every column but the target, so a text column
beside the numbers, one the user never meant as a feature, raised
`ValueError: could not convert string to float`, and a timestamp raised
`DTypePromotionError`. Nothing in the configuration could exclude it,
and the message named neither the column nor a way out. Booleans are
kept, fitting as 0/1. What was left out is printed, so the choice is
visible rather than silent, and this follows the rest of the codebase,
where twenty-four visualization operators drop missing values before
plotting.
A table whose every column but the target is text is narrowed down to
nothing, and a frame of no columns has no dtype for scikit-learn to
read, so numpy raised `at least one array or dtype is required`, naming
neither the columns nor a way out. The narrowing says it itself instead,
naming what it left out and asking for a numeric or boolean column. The
drop is what makes this reachable, so it belongs with it rather than in
a change of its own.
The drop is skipped under the text pipeline, where the
`ColumnTransformer` names the columns it reads and they are exactly the
ones this would drop.
The schema constraints sit on the shared base, so they reach all
fifty-one operators of the Sklearn and Sklearn Training groups. The
required marker for the text columns needs #7642; without it the
configuration is still refused, only unmarked.
The narrowing reaches three more operators, all of which take every
column but the target and descend from `PythonOperatorDescriptor` rather
than one of the sklearn bases. Linear Regression ended the run on a text
column the way the fifty-one used to. Prediction and Testing needed it
for a second reason: a model is fitted by one operator and read by
another, so once the fitting side leaves a column out, the reading side
has to leave out the same one or scikit-learn refuses the frame for
naming a feature it never saw. Prediction reads `feature_names_in_` off
the model rather than re-deriving the rule, since its generated code
holds one Tuple rather than a frame. The one definition now lives in a
trait mixed in wherever a frame is handed to an estimator, so a fourth
operator cannot quietly disagree with the other three.
This is rebased on #7668, which rewrote the same line of the two codegen
templates. That line is resolved the way #7668 asked: the branch this PR
introduces is kept, the one #7668 removed stays removed, and the
condition moved into `dropNonFeatureColumns`, which now emits nothing
when `Count Vectorizer` is on.
### Any related issues, documentation, discussions?
Closes #7643. The marker for the new conditional required comes from
#7642.
### How was this PR tested?
`SklearnClassifierOpDescCodegenSpec` and
`SklearnTrainingOpDescCodegenSpec` already pinned the generated
feature-selection line and were updated to the new one, including that
the text pipeline does not carry it. The three operators added here pin
theirs in their own specs, and removing a narrowing turns the matching
one red. The message for a narrowing that leaves no column is pinned in
`SklearnLinearRegressionOpDescSpec`, which already owns the assertions
on that generated block. `WorkflowOperator/test` passes: 2456 tests.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
---
.../operator/sklearn/SklearnClassifierOpDesc.scala | 1 +
.../operator/sklearn/SklearnFittableColumns.scala | 61 ++++++++++++++++++++++
.../sklearn/SklearnLinearRegressionOpDesc.scala | 3 +-
.../operator/sklearn/SklearnModelOpDesc.scala | 38 +++++++++++++-
.../operator/sklearn/SklearnPredictionOpDesc.scala | 3 ++
.../sklearn/testing/SklearnTestingOpDesc.scala | 4 +-
.../sklearn/training/SklearnTrainingOpDesc.scala | 1 +
.../SklearnClassifierOpDescCodegenSpec.scala | 11 +++-
.../SklearnLinearRegressionOpDescSpec.scala | 21 ++++++++
.../sklearn/SklearnPredictionOpDescSpec.scala | 16 ++++++
.../sklearn/testing/SklearnTestingOpDescSpec.scala | 13 +++++
.../SklearnTrainingOpDescCodegenSpec.scala | 11 +++-
12 files changed, 176 insertions(+), 7 deletions(-)
diff --git
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDesc.scala
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDesc.scala
index 588d717728..a2323e5d19 100644
---
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDesc.scala
+++
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDesc.scala
@@ -46,6 +46,7 @@ abstract class SklearnClassifierOpDesc extends
SklearnModelOpDesc {
| print("Skipped", rows_read - len(table), "of", rows_read,
"rows with missing values")
| Y = table[$target]
| X = table.drop($target, axis=1)
+ |${dropNonFeatureColumns("X", " " * 8)}
$reportMissingKept
| if port == 0:
| self.model = make_pipeline(${vectorizerStage(c =>
pyb"$c".toString)} ${if (
diff --git
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnFittableColumns.scala
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnFittableColumns.scala
new file mode 100644
index 0000000000..97e3cd1d7d
--- /dev/null
+++
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnFittableColumns.scala
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.operator.sklearn
+
+import com.fasterxml.jackson.annotation.JsonIgnore
+
+/** The one definition of which columns an estimator is given.
+ *
+ * A model is fitted by one operator and read by another, so the rule has to
+ * hold across operators rather than within one: a column the fitting side
+ * leaves out is a column the predicting side must leave out too, or
scikit-learn
+ * refuses the frame for naming features it never saw. One definition, mixed
in
+ * wherever a frame is handed to an estimator, is what makes that true by
+ * construction instead of by four copies agreeing.
+ *
+ * Not on [[SklearnModelOpDesc]], where this began: Linear Regression and the
+ * prediction and testing operators descend from `PythonOperatorDescriptor`
+ * directly and could not reach it there.
+ */
+trait SklearnFittableColumns {
+
+ /** Python that narrows `frame` to the columns an estimator can fit, written
at
+ * `indent`.
+ *
+ * A column the user did not mean as a feature, a note beside the numbers,
+ * would otherwise end the run from inside scikit-learn. Booleans are kept:
+ * they fit as 0/1. What was dropped is printed, so the choice is visible
+ * rather than silent.
+ *
+ * A table whose every column but the target is text leaves nothing behind,
+ * and a frame of no columns has no dtype for scikit-learn to read, so it
+ * raises `at least one array or dtype is required` from inside numpy. The
+ * narrowing says so itself instead, naming the columns it left out.
+ */
+ @JsonIgnore
+ protected def narrowToFittableColumns(frame: String, indent: String): String
=
+ s"""${indent}_fittable = $frame.select_dtypes(include=["number", "bool"])
+ |${indent}_ignored = [c for c in $frame.columns if c not in
_fittable.columns]
+ |${indent}if _ignored:
+ |${indent} print("Ignoring columns an estimator cannot fit:",
_ignored)
+ |${indent}if _fittable.columns.empty:
+ |${indent} raise ValueError(f"No column left to fit on: an estimator
cannot fit {_ignored}. Give it a numeric or boolean column.")
+ |${indent}$frame = _fittable""".stripMargin
+}
diff --git
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDesc.scala
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDesc.scala
index e590fa6928..9ea793338e 100644
---
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDesc.scala
+++
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDesc.scala
@@ -29,7 +29,7 @@ import
org.apache.texera.amber.operator.PythonOperatorDescriptor
import
org.apache.texera.amber.operator.metadata.annotations.AutofillAttributeName
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants,
OperatorInfo}
-class SklearnLinearRegressionOpDesc extends PythonOperatorDescriptor {
+class SklearnLinearRegressionOpDesc extends PythonOperatorDescriptor with
SklearnFittableColumns {
@JsonSchemaTitle("Target Attribute")
@JsonPropertyDescription("Attribute in your dataset corresponding to
target.")
@@ -59,6 +59,7 @@ class SklearnLinearRegressionOpDesc extends
PythonOperatorDescriptor {
| print("Skipped", rows_read - len(table), "of", rows_read,
"rows with missing values")
| Y = table[$target]
| X = table.drop($target, axis=1)
+ |${narrowToFittableColumns("X", " " * 8)}
| if port == 0:
| pipeline = make_pipeline(
| PolynomialFeatures(degree=$degree),
diff --git
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnModelOpDesc.scala
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnModelOpDesc.scala
index 74e474f1ac..c771d31aff 100644
---
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnModelOpDesc.scala
+++
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnModelOpDesc.scala
@@ -42,7 +42,35 @@ import
org.apache.texera.amber.operator.metadata.annotations.{
HideAnnotation
}
-abstract class SklearnModelOpDesc extends PythonOperatorDescriptor {
+// `text` names the columns Count Vectorizer tokenizes, so they are string
columns
+// and at least one is required only when that switch is on. Conditional
rather than
+// plain required, so a freshly dropped operator is not flagged for a field it
has
+// no use for.
+@JsonSchemaInject(json = """
+{
+ "attributeTypeRules": {
+ "text": {
+ "enum": ["string"]
+ }
+ },
+ "allOf": [
+ {
+ "if": {
+ "properties": {
+ "countVectorizer": { "const": true }
+ }
+ },
+ "then": {
+ "required": ["text"],
+ "properties": {
+ "text": { "minItems": 1 }
+ }
+ }
+ }
+ ]
+}
+""")
+abstract class SklearnModelOpDesc extends PythonOperatorDescriptor with
SklearnFittableColumns {
@JsonSchemaTitle("Target Attribute")
@JsonPropertyDescription("Attribute in your dataset corresponding to
target.")
@@ -112,6 +140,14 @@ abstract class SklearnModelOpDesc extends
PythonOperatorDescriptor {
.map { case (column, i) => s"""("text$i", CountVectorizer(),
${renderColumn(column)})""" }
.mkString("ColumnTransformer([", ", ", "]),")
+ /** [[SklearnFittableColumns.narrowToFittableColumns]], except under the text
+ * pipeline: there the `ColumnTransformer` names the columns it reads, and
they
+ * are the ones the narrowing would drop.
+ */
+ @JsonIgnore
+ protected def dropNonFeatureColumns(frame: String, indent: String): String =
+ if (countVectorizer) "" else narrowToFittableColumns(frame, indent)
+
@JsonIgnore
def getImportStatements: String
diff --git
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDesc.scala
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDesc.scala
index 0957339ab4..5d8b1b1e7d 100644
---
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDesc.scala
+++
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDesc.scala
@@ -62,6 +62,9 @@ class SklearnPredictionOpDesc extends
PythonOperatorDescriptor {
| input_features = tuple_
| if $groundTruthAttribute != "":
| input_features = input_features.get_partial_tuple([col
for col in tuple_.get_field_names() if col != $groundTruthAttribute])
+ | _fitted = getattr(self.model, "feature_names_in_", None)
+ | if _fitted is not None:
+ | input_features =
input_features.get_partial_tuple(list(_fitted))
| if
Table.from_tuple_likes([input_features]).isna().any(axis=None):
| tuple_[$resultAttribute] = None #keep the row, leave
the result empty
| else:
diff --git
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDesc.scala
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDesc.scala
index 8e633169ee..94319f0936 100644
---
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDesc.scala
+++
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDesc.scala
@@ -24,6 +24,7 @@ import
com.kjetland.jackson.jsonSchema.annotations.JsonSchemaTitle
import org.apache.texera.amber.core.tuple.{AttributeType, Schema}
import org.apache.texera.amber.core.workflow.{InputPort, OutputPort,
PortIdentity}
import org.apache.texera.amber.operator.PythonOperatorDescriptor
+import org.apache.texera.amber.operator.sklearn.SklearnFittableColumns
import org.apache.texera.amber.operator.metadata.annotations.{
AutofillAttributeName,
AutofillAttributeNameOnPort1
@@ -32,7 +33,7 @@ import
org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, Operat
import org.apache.texera.amber.pybuilder.PyStringTypes.EncodableString
import
org.apache.texera.amber.pybuilder.PythonTemplateBuilder.PythonTemplateBuilderStringContext
-class SklearnTestingOpDesc extends PythonOperatorDescriptor {
+class SklearnTestingOpDesc extends PythonOperatorDescriptor with
SklearnFittableColumns {
@JsonProperty(required = true, defaultValue = "false")
@JsonSchemaTitle("Regression")
@JsonPropertyDescription(
@@ -74,6 +75,7 @@ class SklearnTestingOpDesc extends PythonOperatorDescriptor {
| print("Skipped", rows_read - len(table), "of",
rows_read, "rows with missing values")
| Y = table[$target]
| X = table.drop($target, axis=1)
+ |${narrowToFittableColumns("X", " " * 12)}
| predictions = model.predict(X.squeeze())
| if $isRegressionStr:
| tuple_["R2"] = r2_score(Y, predictions)
diff --git
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/training/SklearnTrainingOpDesc.scala
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/training/SklearnTrainingOpDesc.scala
index 3de809dd9f..c51f322a62 100644
---
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/training/SklearnTrainingOpDesc.scala
+++
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/training/SklearnTrainingOpDesc.scala
@@ -46,6 +46,7 @@ class SklearnTrainingOpDesc extends SklearnModelOpDesc {
| print("Skipped", rows_read - len(table), "of", rows_read,
"rows with missing values")
| Y = table[$target]
| X = table.drop($target, axis=1)
+ |${dropNonFeatureColumns("X", " " * 8)}
$reportMissingKept
| model = make_pipeline(${vectorizerStage(c =>
pyb"$c".toString)} ${if (
tfidfTransformer
diff --git
a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDescCodegenSpec.scala
b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDescCodegenSpec.scala
index e964fcca34..0bdd32ec90 100644
---
a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDescCodegenSpec.scala
+++
b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDescCodegenSpec.scala
@@ -68,8 +68,12 @@ class SklearnClassifierOpDescCodegenSpec extends AnyFlatSpec
with Matchers {
code should include("from sklearn.neighbors import KNeighborsClassifier")
code should include(s"Y = table[${decodeExpr("label")}]")
code should include(s"X = table.drop(${decodeExpr("label")}, axis=1)")
- // Feature-column path: X is kept whole, the text attribute is never read.
+ // Feature-column path: every column an estimator can fit is kept, the
rest are
+ // named on the console, and the text attribute is never read.
code should not include "ColumnTransformer("
+ code should include("""_fittable = X.select_dtypes(include=["number",
"bool"])""")
+ code should include("""print("Ignoring columns an estimator cannot fit:",
_ignored)""")
+ code should include("X = _fittable")
code should not include decodeExpr("docs")
normalized(code) should include(
"self.model = make_pipeline( KNeighborsClassifier()).fit(X, Y)"
@@ -81,7 +85,9 @@ class SklearnClassifierOpDescCodegenSpec extends AnyFlatSpec
with Matchers {
it should "select the text column and prepend CountVectorizer when
countVectorizer is on" in {
val code = descriptor(countVectorizer = true).generatePythonCode()
- // ColumnTransformer selects the columns itself, so X stays the whole
frame.
+ // ColumnTransformer selects the columns itself, so X stays the whole
frame, and
+ // narrowing it to the fittable columns would drop the text ones it reads.
+ code should not include "_fittable"
normalized(code) should include(
s"""self.model = make_pipeline(ColumnTransformer([("text0",
CountVectorizer(), ${decodeExpr(
"docs"
@@ -120,6 +126,7 @@ class SklearnClassifierOpDescCodegenSpec extends
AnyFlatSpec with Matchers {
val code = descriptor(tfidfTransformer = true).generatePythonCode()
// Without countVectorizer there is no text-column selection.
code should not include "ColumnTransformer("
+ code should include("X = _fittable")
code should not include decodeExpr("docs")
normalized(code) should include(
"self.model = make_pipeline( TfidfTransformer(),
KNeighborsClassifier()).fit(X, Y)"
diff --git
a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDescSpec.scala
b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDescSpec.scala
index 2c1c02a9eb..d0364e1998 100644
---
a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDescSpec.scala
+++
b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDescSpec.scala
@@ -74,6 +74,27 @@ class SklearnLinearRegressionOpDescSpec extends AnyFlatSpec
with Matchers {
code should include("\"Skipped\"")
}
+ // The operator fits every column but the target, so a text column beside the
+ // numbers would end the run from inside scikit-learn.
+ it should "narrow the features to the columns an estimator can fit" in {
+ val d = new SklearnLinearRegressionOpDesc
+ d.target = "y"
+ val code = d.generatePythonCode()
+ code should include("""_fittable = X.select_dtypes(include=["number",
"bool"])""")
+ code should include("""print("Ignoring columns an estimator cannot fit:",
_ignored)""")
+ code should include("X = _fittable")
+ }
+
+ // Narrowing a table whose every column but the target is text leaves a
frame of
+ // no columns, which numpy answers with `at least one array or dtype is
required`.
+ it should "say so when the narrowing leaves no column at all" in {
+ val d = new SklearnLinearRegressionOpDesc
+ d.target = "y"
+ val code = d.generatePythonCode()
+ code should include("if _fittable.columns.empty:")
+ code should include("No column left to fit on")
+ }
+
"SklearnLinearRegressionOpDesc" should
"round-trip its target through the polymorphic base" in {
val d = new SklearnLinearRegressionOpDesc
diff --git
a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDescSpec.scala
b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDescSpec.scala
index 61e6b1750e..16aec5da13 100644
---
a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDescSpec.scala
+++
b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDescSpec.scala
@@ -138,6 +138,22 @@ class SklearnPredictionOpDescSpec extends AnyFlatSpec with
Matchers {
code should include("str(prediction)")
}
+ // The fitting operators leave out the columns an estimator cannot fit, so
this
+ // side has to leave out the same ones or scikit-learn refuses the frame for
+ // naming features it never saw. Read off the model, which carries what it
was
+ // fitted on, rather than re-deriving a rule that could drift from theirs:
this
+ // path holds one Tuple rather than a frame, where select_dtypes does not
apply.
+ it should "narrow the input features to the ones the model was fitted on" in
{
+ val d = new SklearnPredictionOpDesc
+ d.model = "model"
+ d.resultAttribute = "prediction"
+ d.groundTruthAttribute = "y"
+ val code = d.generatePythonCode()
+ code should include(""""feature_names_in_", None)""")
+ code should include("if _fitted is not None:")
+ code should include("input_features.get_partial_tuple(list(_fitted))")
+ }
+
"SklearnPredictionOpDesc" should
"round-trip its config fields through the polymorphic base" in {
val d = new SklearnPredictionOpDesc
diff --git
a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDescSpec.scala
b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDescSpec.scala
index 68ec31a8d7..8401c6e20b 100644
---
a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDescSpec.scala
+++
b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDescSpec.scala
@@ -89,6 +89,19 @@ class SklearnTestingOpDescSpec extends AnyFlatSpec with
Matchers {
d.generatePythonCode() should include("Table(self.data).dropna()")
}
+ // The scorer reads every column but the target, so it has to leave out what
an
+ // estimator cannot fit for the same reason the fitting operators do, and
leave
+ // out the same columns: a model fitted without them refuses a frame naming
them.
+ it should "narrow the features to the columns an estimator can fit" in {
+ val d = new SklearnTestingOpDesc
+ d.model = "model"
+ d.target = "y"
+ val code = d.generatePythonCode()
+ code should include("""_fittable = X.select_dtypes(include=["number",
"bool"])""")
+ code should include("""print("Ignoring columns an estimator cannot fit:",
_ignored)""")
+ code should include("X = _fittable")
+ }
+
"SklearnTestingOpDesc" should
"round-trip its config fields through the polymorphic base" in {
val d = new SklearnTestingOpDesc
diff --git
a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/training/SklearnTrainingOpDescCodegenSpec.scala
b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/training/SklearnTrainingOpDescCodegenSpec.scala
index 114393d374..23301e18c7 100644
---
a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/training/SklearnTrainingOpDescCodegenSpec.scala
+++
b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/training/SklearnTrainingOpDescCodegenSpec.scala
@@ -68,8 +68,12 @@ class SklearnTrainingOpDescCodegenSpec extends AnyFlatSpec
with Matchers {
code should include("from sklearn.neighbors import KNeighborsClassifier")
code should include(s"Y = table[${decodeExpr("label")}]")
code should include(s"X = table.drop(${decodeExpr("label")}, axis=1)")
- // Feature-column path: X is kept whole, the text attribute is never read.
+ // Feature-column path: every column an estimator can fit is kept, the
rest are
+ // named on the console, and the text attribute is never read.
code should not include "ColumnTransformer("
+ code should include("""_fittable = X.select_dtypes(include=["number",
"bool"])""")
+ code should include("""print("Ignoring columns an estimator cannot fit:",
_ignored)""")
+ code should include("X = _fittable")
code should not include decodeExpr("docs")
normalized(code) should include("make_pipeline(
KNeighborsClassifier()).fit(X, Y)")
code should not include "CountVectorizer()"
@@ -78,7 +82,9 @@ class SklearnTrainingOpDescCodegenSpec extends AnyFlatSpec
with Matchers {
it should "select the text column and prepend CountVectorizer when
countVectorizer is on" in {
val code = descriptor(countVectorizer = true).generatePythonCode()
- // ColumnTransformer selects the columns itself, so X stays the whole
frame.
+ // ColumnTransformer selects the columns itself, so X stays the whole
frame, and
+ // narrowing it to the fittable columns would drop the text ones it reads.
+ code should not include "_fittable"
normalized(code) should include(
s"""make_pipeline(ColumnTransformer([("text0", CountVectorizer(),
${decodeExpr(
"docs"
@@ -117,6 +123,7 @@ class SklearnTrainingOpDescCodegenSpec extends AnyFlatSpec
with Matchers {
val code = descriptor(tfidfTransformer = true).generatePythonCode()
// Without countVectorizer there is no text-column selection.
code should not include "ColumnTransformer("
+ code should include("X = _fittable")
code should not include decodeExpr("docs")
normalized(code) should include(
"make_pipeline( TfidfTransformer(), KNeighborsClassifier()).fit(X, Y)"
