[
https://issues.apache.org/jira/browse/SPARK-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429066#comment-15429066
]
DB Tsai commented on SPARK-17151:
---------------------------------
BTW, not only the zero coefficients issues but also the intercepts will be
negative infinity for those classes which are not seen in the training time.
This will cause some instabilities during the optimization, and we should not
train on those unseen classes. As a result, we need to keep track on what are
the seen classes in the training time, and only optimize the coefficients for
them. Since we know all the possible classes which should be able to be
specified by users as part of the API, in prediction time, we just make them
probability zero.
> Decide how to handle inferring number of classes in Multinomial logistic
> regression
> -----------------------------------------------------------------------------------
>
> Key: SPARK-17151
> URL: https://issues.apache.org/jira/browse/SPARK-17151
> Project: Spark
> Issue Type: Sub-task
> Components: ML, MLlib
> Reporter: Seth Hendrickson
> Priority: Minor
>
> This JIRA is to discuss how the number of label classes should be inferred in
> multinomial logistic regression. Currently, MLOR checks the dataframe
> metadata and if the number of classes is not specified then it uses the
> maximum value seen in the label column. If the labels are not properly
> indexed, then this can cause a large number of zero coefficients and
> potentially produce instabilities in model training.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
