[
https://issues.apache.org/jira/browse/SPARK-7568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiangrui Meng updated SPARK-7568:
---------------------------------
Description:
`bin/spark-submit
examples/src/main/python/ml/simple_text_classification_pipeline.py`
{code}
Row(id=4, text=u'spark i j k', words=[u'spark', u'i', u'j', u'k'],
features=SparseVector(262144, {105: 1.0, 106: 1.0, 107: 1.0, 62173: 1.0}),
rawPrediction=DenseVector([0.1629, -0.1629]), probability=DenseVector([0.5406,
0.4594]), prediction=0.0)
Row(id=5, text=u'l m n', words=[u'l', u'm', u'n'],
features=SparseVector(262144, {108: 1.0, 109: 1.0, 110: 1.0}),
rawPrediction=DenseVector([2.6407, -2.6407]), probability=DenseVector([0.9334,
0.0666]), prediction=0.0)
Row(id=6, text=u'mapreduce spark', words=[u'mapreduce', u'spark'],
features=SparseVector(262144, {62173: 1.0, 140738: 1.0}),
rawPrediction=DenseVector([1.2651, -1.2651]), probability=DenseVector([0.7799,
0.2201]), prediction=0.0)
Row(id=7, text=u'apache hadoop', words=[u'apache', u'hadoop'],
features=SparseVector(262144, {128334: 1.0, 134181: 1.0}),
rawPrediction=DenseVector([3.7429, -3.7429]), probability=DenseVector([0.9769,
0.0231]), prediction=0.0)
{code}
All predictions are 0, while some should be one based on the probability. It
seems to be an issue with regularization.
was:
`bin/spark-submit
examples/src/main/python/ml/simple_text_classification_pipeline.py`
{code}
Row(id=4, text=u'spark i j k', words=[u'spark', u'i', u'j', u'k'],
features=SparseVector(262144, {105: 1.0, 106: 1.0, 107: 1.0, 62173: 1.0}),
rawPrediction=DenseVector([0.1629, -0.1629]), probability=DenseVector([0.5406,
0.4594]), prediction=0.0)
Row(id=5, text=u'l m n', words=[u'l', u'm', u'n'],
features=SparseVector(262144, {108: 1.0, 109: 1.0, 110: 1.0}),
rawPrediction=DenseVector([2.6407, -2.6407]), probability=DenseVector([0.9334,
0.0666]), prediction=0.0)
Row(id=6, text=u'mapreduce spark', words=[u'mapreduce', u'spark'],
features=SparseVector(262144, {62173: 1.0, 140738: 1.0}),
rawPrediction=DenseVector([1.2651, -1.2651]), probability=DenseVector([0.7799,
0.2201]), prediction=0.0)
Row(id=7, text=u'apache hadoop', words=[u'apache', u'hadoop'],
features=SparseVector(262144, {128334: 1.0, 134181: 1.0}),
rawPrediction=DenseVector([3.7429, -3.7429]), probability=DenseVector([0.9769,
0.0231]), prediction=0.0)
{code}
All predictions are 0, while some should be one based on the probability.
> ml.LogisticRegression doesn't output the right prediction
> ---------------------------------------------------------
>
> Key: SPARK-7568
> URL: https://issues.apache.org/jira/browse/SPARK-7568
> Project: Spark
> Issue Type: Bug
> Components: ML
> Affects Versions: 1.4.0
> Reporter: Xiangrui Meng
> Assignee: DB Tsai
> Priority: Blocker
>
> `bin/spark-submit
> examples/src/main/python/ml/simple_text_classification_pipeline.py`
> {code}
> Row(id=4, text=u'spark i j k', words=[u'spark', u'i', u'j', u'k'],
> features=SparseVector(262144, {105: 1.0, 106: 1.0, 107: 1.0, 62173: 1.0}),
> rawPrediction=DenseVector([0.1629, -0.1629]),
> probability=DenseVector([0.5406, 0.4594]), prediction=0.0)
> Row(id=5, text=u'l m n', words=[u'l', u'm', u'n'],
> features=SparseVector(262144, {108: 1.0, 109: 1.0, 110: 1.0}),
> rawPrediction=DenseVector([2.6407, -2.6407]),
> probability=DenseVector([0.9334, 0.0666]), prediction=0.0)
> Row(id=6, text=u'mapreduce spark', words=[u'mapreduce', u'spark'],
> features=SparseVector(262144, {62173: 1.0, 140738: 1.0}),
> rawPrediction=DenseVector([1.2651, -1.2651]),
> probability=DenseVector([0.7799, 0.2201]), prediction=0.0)
> Row(id=7, text=u'apache hadoop', words=[u'apache', u'hadoop'],
> features=SparseVector(262144, {128334: 1.0, 134181: 1.0}),
> rawPrediction=DenseVector([3.7429, -3.7429]),
> probability=DenseVector([0.9769, 0.0231]), prediction=0.0)
> {code}
> All predictions are 0, while some should be one based on the probability. It
> seems to be an issue with regularization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
