carloea2 commented on code in PR #8368:
URL: https://github.com/apache/texera/pull/8368#discussion_r4079828164
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDesc.scala:
##########
@@ -99,4 +100,71 @@ class SklearnPredictionOpDesc extends
PythonOperatorDescriptor {
.add(resultAttribute, resultType)
)
}
+
+ /** Python that narrows `X` to the columns the model was fitted on.
+ *
+ * The fitting side leaves out the columns an estimator cannot fit, so this
+ * side has to leave out the same ones or scikit-learn refuses the frame for
+ * naming features it never saw. Read off the model rather than re-derived:
+ * what it was fitted on is a fact it carries, and asking it cannot drift
from
+ * whatever rule the fitting operator applied.
+ */
+ private val narrowToFittedFeatures: String =
+ """_fitted = getattr(model, "feature_names_in_", None)
+ |if _fitted is not None:
+ | X = X[list(_fitted)]""".stripMargin
+
+ /** Python that predicts on the rows whose features are all present.
+ *
+ * The executor takes a row at a time and leaves the result empty where a
+ * feature is missing, so the row keeps its place. One call over the whole
+ * frame cannot do that: scikit-learn ends the run on the first missing
value
+ * rather than leaving a gap. So predict on the complete rows and write
those
+ * answers back where they came from. A frame with nothing missing is
+ * assigned in one piece, which keeps the type `predict` returned.
+ */
+ private def predictLeavingIncompleteRowsEmpty(resultLit: String, asText:
Boolean): String = {
+ val predicted =
+ if (asText) "[str(_p) for _p in model.predict(X[_complete])]"
+ else "model.predict(X[_complete]).tolist()"
+ s"""_complete = X.notna().all(axis=1)
+ |_predicted = $predicted if _complete.any() else []
+ |if _complete.all():
+ | out1df[$resultLit] = _predicted
+ |else:
+ | out1df[$resultLit] = None
+ | out1df.loc[_complete, $resultLit] = _predicted""".stripMargin
+ }
+
+ /** Python that takes the model the executor predicts with.
+ *
+ * The executor keeps the model of every row the model port hands it, each
one
+ * overwriting the last, so the model it predicts with is the one on the
final
+ * row. Reading the first row instead would answer with a different model
+ * whenever that port carries more than one.
+ */
+ private def takeTheModelTheExecutorKeeps(modelLit: String): String =
+ s"model = in1df[$modelLit].iloc[-1]"
Review Comment:
When both input ports are empty, this raises IndexError on iloc[-1]. The
native row-driven operator makes no prediction calls and emits no rows. Empty
data with a supplied model passes as a control. Please return an empty result
with the declared columns before accessing the model when there are no data
rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
