Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/5245#issuecomment-95810160
@mengxr I do some tests on these two versions, here is the result log: (You
can see my code
[here](https://github.com/yinxusen/spark/blob/PerformanceTest-5894/mllib/src/main/scala/org/apache/spark/ml/feature/PolynomialMapper.scala).)
```bash
sbt "mllib/run-main org.apache.spark.ml.feature.PolynomialMapper"
2>&1>test.log
```
> [info] Testing number of data 1024
[info] Testing dataset degree: 2 mapper: PolynomialMapper-V1
name: denseData
[info] Elapsed time: 48.591317ms
[info] Testing dataset degree: 2 mapper: PolynomialMapper-V1
name: sparseData
[info] Elapsed time: 43.113877ms
[info] Testing dataset degree: 2 mapper: PolynomialMapper-V2
name: denseData
[info] Elapsed time: 38.518744ms
[info] Testing dataset degree: 2 mapper: PolynomialMapper-V2
name: sparseData
[info] Elapsed time: 36.946037ms
[info] Testing dataset degree: 3 mapper: PolynomialMapper-V1
name: denseData
[info] Elapsed time: 34.615637ms
[info] Testing dataset degree: 3 mapper: PolynomialMapper-V1
name: sparseData
[info] Elapsed time: 39.327571ms
[info] Testing dataset degree: 3 mapper: PolynomialMapper-V2
name: denseData
[info] Elapsed time: 35.640954ms
[info] Testing dataset degree: 3 mapper: PolynomialMapper-V2
name: sparseData
[info] Elapsed time: 38.740797ms
[info] Testing dataset degree: 5 mapper: PolynomialMapper-V1
name: denseData
[info] Elapsed time: 37.757011ms
[info] Testing dataset degree: 5 mapper: PolynomialMapper-V1
name: sparseData
[info] Elapsed time: 39.291329ms
[info] Testing dataset degree: 5 mapper: PolynomialMapper-V2
name: denseData
[info] Elapsed time: 34.665687ms
[info] Testing dataset degree: 5 mapper: PolynomialMapper-V2
name: sparseData
[info] Elapsed time: 37.758357ms
[info] Testing dataset degree: 10 mapper: PolynomialMapper-V1
name: denseData
[info] Elapsed time: 33.307436ms
[info] Testing dataset degree: 10 mapper: PolynomialMapper-V1
name: sparseData
[info] Elapsed time: 37.231837ms
[info] Testing dataset degree: 10 mapper: PolynomialMapper-V2
name: denseData
[info] Elapsed time: 34.794309ms
[info] Testing dataset degree: 10 mapper: PolynomialMapper-V2
name: sparseData
[info] Elapsed time: 37.112773ms
> [info] Testing number of data 10240
[info] Testing dataset degree: 2 mapper: PolynomialMapper-V1
name: denseData
[info] Elapsed time: 76.447725ms
[info] Testing dataset degree: 2 mapper: PolynomialMapper-V1
name: sparseData
[info] Elapsed time: 98.351862ms
[info] Testing dataset degree: 2 mapper: PolynomialMapper-V2
name: denseData
[info] Elapsed time: 76.17611ms
[info] Testing dataset degree: 2 mapper: PolynomialMapper-V2
name: sparseData
[info] Elapsed time: 99.099883ms
[info] Testing dataset degree: 3 mapper: PolynomialMapper-V1
name: denseData
[info] Elapsed time: 76.661511ms
[info] Testing dataset degree: 3 mapper: PolynomialMapper-V1
name: sparseData
[info] Elapsed time: 99.442798ms
[info] Testing dataset degree: 3 mapper: PolynomialMapper-V2
name: denseData
[info] Elapsed time: 76.607076ms
[info] Testing dataset degree: 3 mapper: PolynomialMapper-V2
name: sparseData
[info] Elapsed time: 99.722276ms
[info] Testing dataset degree: 5 mapper: PolynomialMapper-V1
name: denseData
[info] Elapsed time: 76.337466ms
[info] Testing dataset degree: 5 mapper: PolynomialMapper-V1
name: sparseData
[info] Elapsed time: 99.550001ms
[info] Testing dataset degree: 5 mapper: PolynomialMapper-V2
name: denseData
[info] Elapsed time: 76.633637ms
[info] Testing dataset degree: 5 mapper: PolynomialMapper-V2
name: sparseData
[info] Elapsed time: 98.995122ms
[info] Testing dataset degree: 10 mapper: PolynomialMapper-V1
name: denseData
[info] Elapsed time: 77.281723ms
[info] Testing dataset degree: 10 mapper: PolynomialMapper-V1
name: sparseData
[info] Elapsed time: 100.623104ms
[info] Testing dataset degree: 10 mapper: PolynomialMapper-V2
name: denseData
[info] Elapsed time: 76.444839ms
[info] Testing dataset degree: 10 mapper: PolynomialMapper-V2
name: sparseData
[info] Elapsed time: 99.626543ms
> [info] Testing number of data 102400
[info] Testing dataset degree: 2 mapper: PolynomialMapper-V1
name: denseData
[info] Elapsed time: 459.402129ms
[info] Testing dataset degree: 2 mapper: PolynomialMapper-V1
name: sparseData
[info] Elapsed time: 744.510455ms
[info] Testing dataset degree: 2 mapper: PolynomialMapper-V2
name: denseData
[info] Elapsed time: 462.813174ms
[info] Testing dataset degree: 2 mapper: PolynomialMapper-V2
name: sparseData
[info] Elapsed time: 745.74096ms
[info] Testing dataset degree: 3 mapper: PolynomialMapper-V1
name: denseData
[info] Elapsed time: 465.842966ms
[info] Testing dataset degree: 3 mapper: PolynomialMapper-V1
name: sparseData
[info] Elapsed time: 740.870635ms
[info] Testing dataset degree: 3 mapper: PolynomialMapper-V2
name: denseData
[info] Elapsed time: 456.569887ms
[info] Testing dataset degree: 3 mapper: PolynomialMapper-V2
name: sparseData
[info] Elapsed time: 740.952329ms
[info] Testing dataset degree: 5 mapper: PolynomialMapper-V1
name: denseData
[info] Elapsed time: 461.351083ms
[info] Testing dataset degree: 5 mapper: PolynomialMapper-V1
name: sparseData
[info] Elapsed time: 747.656513ms
[info] Testing dataset degree: 5 mapper: PolynomialMapper-V2
name: denseData
[info] Elapsed time: 461.394609ms
[info] Testing dataset degree: 5 mapper: PolynomialMapper-V2
name: sparseData
[info] Elapsed time: 736.373809ms
[info] Testing dataset degree: 10 mapper: PolynomialMapper-V1
name: denseData
[info] Elapsed time: 459.730604ms
[info] Testing dataset degree: 10 mapper: PolynomialMapper-V1
name: sparseData
[info] Elapsed time: 738.605976ms
[info] Testing dataset degree: 10 mapper: PolynomialMapper-V2
name: denseData
[info] Elapsed time: 461.859974ms
[info] Testing dataset degree: 10 mapper: PolynomialMapper-V2
name: sparseData
[info] Elapsed time: 744.532121ms
We can see that the two methods have similar performances.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
