GitHub user yinxusen opened a pull request:
https://github.com/apache/spark/pull/11835
[SPARK-13951] Add nested Pipeline load/save supports in PySpark
## What changes were proposed in this pull request?
https://issues.apache.org/jira/browse/SPARK-13951
1. Scala side `Pipeline` changes:
* Change `Param[Array[PipelineStage]]` to `StageArrayParam` to support
Java competible function.
2. Python side changes:
* wrapper: Add a `JavaConvertible` to support those stages that are not a
`JavaWrapper`.
* wrapper: Add a `ConvertUtil` to support the Python-Scala converting for
both `JavaWrapper` and `JavaConvertible`.
* pipeline: `Pipeline` and `PipelineModel` now extend from
`JavaConvertible`.
* pipeline: `PipelineMLReader`, `PipelineMLWriter`,
`PipelineModelMLReader`, and `PipelineModelMLWriter` now use `ConvertUtils`.
## How was this patch tested?
Test with Python unit test for both pipeline save/load and nested-pipeline
save/load.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yinxusen/spark SPARK-13951
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11835.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11835
----
commit 459c073610608738ed9102f57a88e31a38c87db3
Author: Xusen Yin <[email protected]>
Date: 2016-03-17T06:57:23Z
relax read side
commit 6f035ca2f4c67ad69ee702aaf7894dda60b38add
Author: Xusen Yin <[email protected]>
Date: 2016-03-17T06:58:27Z
relax write side
commit d13bf3a499c83f8841982f1ec83ed2273d204caf
Author: Xusen Yin <[email protected]>
Date: 2016-03-18T00:02:38Z
add test
commit 9e9bf2f1f161f6863e2ca80e524f3dc84b25ccf4
Author: Xusen Yin <[email protected]>
Date: 2016-03-18T07:19:25Z
version 1, fix nested pipeline load/save
commit ce202a2db73886414e79f16e17af8231fa9751a2
Author: Xusen Yin <[email protected]>
Date: 2016-03-18T22:00:00Z
another step to add JavaConvertible
commit d0ae8e2aaca91019f98180933b8e236ac2e891c7
Author: Xusen Yin <[email protected]>
Date: 2016-03-18T22:39:53Z
fix all
commit 4a009902025d0ce954e049fd73fad73547757fff
Author: Xusen Yin <[email protected]>
Date: 2016-03-18T22:56:16Z
merge with master
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
