Github user dputler commented on the pull request:
https://github.com/apache/spark/pull/7987#issuecomment-133112345
That actually doesn't deal with the scoring issue. What happens when new
data to be predicted from an existing model has a more frequent category in a
categorical variable than was the case in the training data? What happens if
this is included in a Spark Streaming scoring process when the batch size might
be one? As before, the frequency base indexing works for estimation, but will
cause heartburn in many cases when trying to predict new data with an existing
model.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
