[jira] [Commented] (SPARK-6348) Enable useFeatureScaling in SVMWithSGD
[ https://issues.apache.org/jira/browse/SPARK-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364310#comment-14364310 ] tanyinyan commented on SPARK-6348: -- Yes,I use a one-hot encoding before SVM , which is the 'sparsed before SVM ' exactly means ：） Enable useFeatureScaling in SVMWithSGD -- Key: SPARK-6348 URL: https://issues.apache.org/jira/browse/SPARK-6348 Project: Spark Issue Type: Improvement Components: MLlib Affects Versions: 1.2.1 Reporter: tanyinyan Priority: Minor Original Estimate: 2h Remaining Estimate: 2h Currently,useFeatureScaling are set to false by default in class GeneralizedLinearAlgorithm, and it is only enabled in LogisticRegressionWithLBFGS. SVMWithSGD class is a private class,train methods are provide in SVMWithSGD object. So there is no way to set useFeatureScaling when using SVM. I am using SVM on dataset(https://www.kaggle.com/c/avazu-ctr-prediction/data), train on the first day's dataset(ignore field id/device_id/device_ip, all remaining fields are concidered as categorical variable, and sparsed before SVM) and predict on the same data with threshold cleared, the predict result are all negative. Then i set useFeatureScaling to true, the predict result are normal(including negative and positive result) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6349) Add probability estimates in SVMModel predict result
[ https://issues.apache.org/jira/browse/SPARK-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364361#comment-14364361 ] tanyinyan commented on SPARK-6349: -- Yes, this doesn't solve the problem of picking which threshold. But a raw margin usually has no fixed boundary(as i tested above, output margin are all negative),but a probability threshold has. So it's more convenient to pick a good threshold , right? Add probability estimates in SVMModel predict result Key: SPARK-6349 URL: https://issues.apache.org/jira/browse/SPARK-6349 Project: Spark Issue Type: New Feature Components: MLlib Affects Versions: 1.2.1 Reporter: tanyinyan Original Estimate: 168h Remaining Estimate: 168h In SVMModel, predictPoint method output raw margin(threshold not set) or 1/0 label(threshold set). when SVM are used as a classifier, it's hard to find a good threshold,and the raw margin is hard to understand. when I am using SVM on dataset(https://www.kaggle.com/c/avazu-ctr-prediction/data), train on the first day's dataset(ignore field id/device_id/device_ip, all remaining fields are concidered as categorical variable, and sparsed before SVM) and predict on the same data with threshold cleared, the predict result are all negative. I have to set threshold to -1 to get a reasonable confusion matrix. So, I suggest to provide probability predict result in SVMModel as in libSVM(Platt's binary SVM Probablistic Output) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6348) Enable useFeatureScaling in SVMWithSGD
[ https://issues.apache.org/jira/browse/SPARK-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364309#comment-14364309 ] tanyinyan commented on SPARK-6348: -- Yes,I use a one-hot encoding before SVM , which is the 'sparsed before SVM ' exactly means ：） Enable useFeatureScaling in SVMWithSGD -- Key: SPARK-6348 URL: https://issues.apache.org/jira/browse/SPARK-6348 Project: Spark Issue Type: Improvement Components: MLlib Affects Versions: 1.2.1 Reporter: tanyinyan Priority: Minor Original Estimate: 2h Remaining Estimate: 2h Currently,useFeatureScaling are set to false by default in class GeneralizedLinearAlgorithm, and it is only enabled in LogisticRegressionWithLBFGS. SVMWithSGD class is a private class,train methods are provide in SVMWithSGD object. So there is no way to set useFeatureScaling when using SVM. I am using SVM on dataset(https://www.kaggle.com/c/avazu-ctr-prediction/data), train on the first day's dataset(ignore field id/device_id/device_ip, all remaining fields are concidered as categorical variable, and sparsed before SVM) and predict on the same data with threshold cleared, the predict result are all negative. Then i set useFeatureScaling to true, the predict result are normal(including negative and positive result) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6348) Enable useFeatureScaling in SVMWithSGD
[ https://issues.apache.org/jira/browse/SPARK-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364311#comment-14364311 ] tanyinyan commented on SPARK-6348: -- Yes,I use a one-hot encoding before SVM , which is the 'sparsed before SVM ' exactly means ：） Enable useFeatureScaling in SVMWithSGD -- Key: SPARK-6348 URL: https://issues.apache.org/jira/browse/SPARK-6348 Project: Spark Issue Type: Improvement Components: MLlib Affects Versions: 1.2.1 Reporter: tanyinyan Priority: Minor Original Estimate: 2h Remaining Estimate: 2h Currently,useFeatureScaling are set to false by default in class GeneralizedLinearAlgorithm, and it is only enabled in LogisticRegressionWithLBFGS. SVMWithSGD class is a private class,train methods are provide in SVMWithSGD object. So there is no way to set useFeatureScaling when using SVM. I am using SVM on dataset(https://www.kaggle.com/c/avazu-ctr-prediction/data), train on the first day's dataset(ignore field id/device_id/device_ip, all remaining fields are concidered as categorical variable, and sparsed before SVM) and predict on the same data with threshold cleared, the predict result are all negative. Then i set useFeatureScaling to true, the predict result are normal(including negative and positive result) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6348) Enable useFeatureScaling in SVMWithSGD
[ https://issues.apache.org/jira/browse/SPARK-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364313#comment-14364313 ] tanyinyan commented on SPARK-6348: -- Yes,I use a one-hot encoding before SVM , which is the 'sparsed before SVM ' exactly means ：） Enable useFeatureScaling in SVMWithSGD -- Key: SPARK-6348 URL: https://issues.apache.org/jira/browse/SPARK-6348 Project: Spark Issue Type: Improvement Components: MLlib Affects Versions: 1.2.1 Reporter: tanyinyan Priority: Minor Original Estimate: 2h Remaining Estimate: 2h Currently,useFeatureScaling are set to false by default in class GeneralizedLinearAlgorithm, and it is only enabled in LogisticRegressionWithLBFGS. SVMWithSGD class is a private class,train methods are provide in SVMWithSGD object. So there is no way to set useFeatureScaling when using SVM. I am using SVM on dataset(https://www.kaggle.com/c/avazu-ctr-prediction/data), train on the first day's dataset(ignore field id/device_id/device_ip, all remaining fields are concidered as categorical variable, and sparsed before SVM) and predict on the same data with threshold cleared, the predict result are all negative. Then i set useFeatureScaling to true, the predict result are normal(including negative and positive result) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6348) Enable useFeatureScaling in SVMWithSGD
[ https://issues.apache.org/jira/browse/SPARK-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364312#comment-14364312 ] tanyinyan commented on SPARK-6348: -- Yes,I use a one-hot encoding before SVM , which is the 'sparsed before SVM ' exactly means ：） Enable useFeatureScaling in SVMWithSGD -- Key: SPARK-6348 URL: https://issues.apache.org/jira/browse/SPARK-6348 Project: Spark Issue Type: Improvement Components: MLlib Affects Versions: 1.2.1 Reporter: tanyinyan Priority: Minor Original Estimate: 2h Remaining Estimate: 2h Currently,useFeatureScaling are set to false by default in class GeneralizedLinearAlgorithm, and it is only enabled in LogisticRegressionWithLBFGS. SVMWithSGD class is a private class,train methods are provide in SVMWithSGD object. So there is no way to set useFeatureScaling when using SVM. I am using SVM on dataset(https://www.kaggle.com/c/avazu-ctr-prediction/data), train on the first day's dataset(ignore field id/device_id/device_ip, all remaining fields are concidered as categorical variable, and sparsed before SVM) and predict on the same data with threshold cleared, the predict result are all negative. Then i set useFeatureScaling to true, the predict result are normal(including negative and positive result) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-6348) Enable useFeatureScaling in SVMWithSGD
[ https://issues.apache.org/jira/browse/SPARK-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tanyinyan updated SPARK-6348: - Comment: was deleted (was: Yes,I use a one-hot encoding before SVM , which is the 'sparsed before SVM ' exactly means ：）) Enable useFeatureScaling in SVMWithSGD -- Key: SPARK-6348 URL: https://issues.apache.org/jira/browse/SPARK-6348 Project: Spark Issue Type: Improvement Components: MLlib Affects Versions: 1.2.1 Reporter: tanyinyan Priority: Minor Original Estimate: 2h Remaining Estimate: 2h Currently,useFeatureScaling are set to false by default in class GeneralizedLinearAlgorithm, and it is only enabled in LogisticRegressionWithLBFGS. SVMWithSGD class is a private class,train methods are provide in SVMWithSGD object. So there is no way to set useFeatureScaling when using SVM. I am using SVM on dataset(https://www.kaggle.com/c/avazu-ctr-prediction/data), train on the first day's dataset(ignore field id/device_id/device_ip, all remaining fields are concidered as categorical variable, and sparsed before SVM) and predict on the same data with threshold cleared, the predict result are all negative. Then i set useFeatureScaling to true, the predict result are normal(including negative and positive result) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-6348) Enable useFeatureScaling in SVMWithSGD
[ https://issues.apache.org/jira/browse/SPARK-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tanyinyan updated SPARK-6348: - Comment: was deleted (was: Yes,I use a one-hot encoding before SVM , which is the 'sparsed before SVM ' exactly means ：）) Enable useFeatureScaling in SVMWithSGD -- Key: SPARK-6348 URL: https://issues.apache.org/jira/browse/SPARK-6348 Project: Spark Issue Type: Improvement Components: MLlib Affects Versions: 1.2.1 Reporter: tanyinyan Priority: Minor Original Estimate: 2h Remaining Estimate: 2h Currently,useFeatureScaling are set to false by default in class GeneralizedLinearAlgorithm, and it is only enabled in LogisticRegressionWithLBFGS. SVMWithSGD class is a private class,train methods are provide in SVMWithSGD object. So there is no way to set useFeatureScaling when using SVM. I am using SVM on dataset(https://www.kaggle.com/c/avazu-ctr-prediction/data), train on the first day's dataset(ignore field id/device_id/device_ip, all remaining fields are concidered as categorical variable, and sparsed before SVM) and predict on the same data with threshold cleared, the predict result are all negative. Then i set useFeatureScaling to true, the predict result are normal(including negative and positive result) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-6348) Enable useFeatureScaling in SVMWithSGD
[ https://issues.apache.org/jira/browse/SPARK-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tanyinyan updated SPARK-6348: - Comment: was deleted (was: Yes,I use a one-hot encoding before SVM , which is the 'sparsed before SVM ' exactly means ：）) Enable useFeatureScaling in SVMWithSGD -- Key: SPARK-6348 URL: https://issues.apache.org/jira/browse/SPARK-6348 Project: Spark Issue Type: Improvement Components: MLlib Affects Versions: 1.2.1 Reporter: tanyinyan Priority: Minor Original Estimate: 2h Remaining Estimate: 2h Currently,useFeatureScaling are set to false by default in class GeneralizedLinearAlgorithm, and it is only enabled in LogisticRegressionWithLBFGS. SVMWithSGD class is a private class,train methods are provide in SVMWithSGD object. So there is no way to set useFeatureScaling when using SVM. I am using SVM on dataset(https://www.kaggle.com/c/avazu-ctr-prediction/data), train on the first day's dataset(ignore field id/device_id/device_ip, all remaining fields are concidered as categorical variable, and sparsed before SVM) and predict on the same data with threshold cleared, the predict result are all negative. Then i set useFeatureScaling to true, the predict result are normal(including negative and positive result) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-6348) Enable useFeatureScaling in SVMWithSGD
[ https://issues.apache.org/jira/browse/SPARK-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tanyinyan updated SPARK-6348: - Comment: was deleted (was: Yes,I use a one-hot encoding before SVM , which is the 'sparsed before SVM ' exactly means ：）) Enable useFeatureScaling in SVMWithSGD -- Key: SPARK-6348 URL: https://issues.apache.org/jira/browse/SPARK-6348 Project: Spark Issue Type: Improvement Components: MLlib Affects Versions: 1.2.1 Reporter: tanyinyan Priority: Minor Original Estimate: 2h Remaining Estimate: 2h Currently,useFeatureScaling are set to false by default in class GeneralizedLinearAlgorithm, and it is only enabled in LogisticRegressionWithLBFGS. SVMWithSGD class is a private class,train methods are provide in SVMWithSGD object. So there is no way to set useFeatureScaling when using SVM. I am using SVM on dataset(https://www.kaggle.com/c/avazu-ctr-prediction/data), train on the first day's dataset(ignore field id/device_id/device_ip, all remaining fields are concidered as categorical variable, and sparsed before SVM) and predict on the same data with threshold cleared, the predict result are all negative. Then i set useFeatureScaling to true, the predict result are normal(including negative and positive result) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-6349) Add probability estimates in SVMModel predict result
tanyinyan created SPARK-6349: Summary: Add probability estimates in SVMModel predict result Key: SPARK-6349 URL: https://issues.apache.org/jira/browse/SPARK-6349 Project: Spark Issue Type: New Feature Components: MLlib Affects Versions: 1.2.1 Reporter: tanyinyan In SVMModel, predictPoint method output raw margin(threshold not set) or 1/0 label(threshold set). when SVM are used as a classifier, it's hard to find a good threshold,and the raw margin is hard to understand. when I am using SVM on dataset(https://www.kaggle.com/c/avazu-ctr-prediction/data), train on the first day's dataset(ignore field id/device_id/device_ip, all remaining fields are concidered as categorical variable, and sparsed before SVM) and predict on the same data with threshold cleared, the predict result are all negative. I have to set threshold to -1 to get a reasonable confusion matrix. So, I suggest to provide probability predict result in SVMModel as in libSVM(Platt's binary SVM Probablistic Output) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-6348) Enable useFeatureScaling in SVMWithSGD
tanyinyan created SPARK-6348: Summary: Enable useFeatureScaling in SVMWithSGD Key: SPARK-6348 URL: https://issues.apache.org/jira/browse/SPARK-6348 Project: Spark Issue Type: Improvement Components: MLlib Affects Versions: 1.2.1 Reporter: tanyinyan Priority: Minor Currently,useFeatureScaling are set to false by default in class GeneralizedLinearAlgorithm, and it is only enabled in LogisticRegressionWithLBFGS. SVMWithSGD class is a private class,train methods are provide in SVMWithSGD object. So there is no way to set useFeatureScaling when using SVM. I am using SVM on dataset(https://www.kaggle.com/c/avazu-ctr-prediction/data), train on the first day's dataset(ignore field id/device_id/device_ip, all remaining fields are concidered as categorical variable, and sparsed before SVM) and predict on the same data with threshold cleared, the predict result are all negative. Then i set useFeatureScaling to true, the predict result are normal(including negative and positive result) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
